ETH Documentation
From MobiComp
ETH Application - Some Documentation
From: "A Smart Museum installation in the Stadsmuseum in Stockholm - From Visitor Guides to Museum Management" by Ryan et al.
"The ETH Object Recognition Guide is implemented on a Tablet PC using a conventional USB webcam to acquire images and to present users with a viewfinder window. When a user wants to get information about an exhibit, the information button on the tablet is pressed when the exhibit is within the viewfinder window. If the exhibit is recognised, the corresponding information is displayed. The MobiComp components used are the camera Tracker, the ETH Image recognition aggregator and the URL display listener. "
In terms of its Mobicomp structure, the application makes use of several mobicomp components working together, as depicted in the image below:
A tracker, called ETHCameraTracker, is used to capture images from the outside world (in this case depicting museum exhibits). The resulting image events are then sent to the Mobicomp infrastructure and propagated to whoever is listening. Another tracker implementation, called ETHImageTracker listens for these images events, and if one is found, tries to match it to a known exhibit making use of the ETH object recognizer algorithm. If a good enough match is made, this tracker then generates a Mobicomp beacon event which (when spotted by the Mobicomp BeaconListener) displays the matching information webpage about the particular exhibit which has just been recognized.
The following sections describe the individual bits that make up the ETH Museum Guide application in more detail. All source code is in the org.mobicomp.eth package and can be found in the src/j2se branch.
1. ETHImageTracker
The mobicomp tracker that waits for image events. If the image is an eth image, it makes a surf file out of it and then compares it to a directory of resident surf files (usually denoting exhibits that we are trying to recognize). If a good enough match is made, a webpage about the exhibit is launched using a beacon html event.
2. ETHCameraTrackerGUI
The mobicomp tracker that grabs a video device, dumps its output to a Swing GUI and prompts a user to take a snapshot, which will then be sent to mobicomp as an image event. The bulk of this app is stolen from TimCam [1], an open source java video device streamer.
3. Code putting it all togther.
The ETHRunner makes sure that the above trackers work together and look like one application. (It also handles the special case where a snapshot is not recognized <- this must be there, else causes inconsistencies in the overall mobicomp model - we do not want 'no result' beacons events floating around.)
4. SurfFileMaker
TIMMainFrameSurfMaker is a runnable class cut out of the original app that only takes images and turns them into surf files in a specific directory. This is used when seting up your 'database' of items to compare against -> (directory of resident surf files usually denoting exhibits that we are trying to recognize).
5. Linux (Debian) only surf.ln
An executable that makes surf files out of specific image files. Used by ETHImageTracker and the SurfFileMaker. Be aware of the linux only bit.
Design Notes:
-ETHImageTracker is reasonably independent of anything else. With a minimal re-write it could be used with anything that gives it the right images and context events.
-ETHCameraTracjerGUI is a bit of a beast, as it links to all the TimCam classes that handle
video (some of them static..). The Tracker class itself is not too messy though.
-ETHRunner is there because after the ETHCameraTrackerGUI takes a snapshot (and that shot is
then decoded by ETHImageTracker) ETHImageTracker must then tell ETHCameraTracker that it is
good to go again. Recognizing objects takes a long time (can take upwards of 10 seconds)
so after a shot is taken the app must make sure not to try again until after the
ETHImageTracker is done.
Hence the 2 trackers must have references to each other. (Or at least ETHImageTracker must be able to somehow communicate to ETHCameraTrackerGUI when it is done.) That is why the runner is here. Perhaps a html beacon event could be used to achieve this instead? To think about.
How to run:
To run this application, invoke the ETHRunner, for example like so: java -classpath /..../mobicomp/dist/linux/context.jar org.mobicomp.eth.ETHRunner
To run the SurfFileMaker, invoke it, for example like so: java -classpath /..../mobicomp/dist/linux/context.jar org.mobicomp.eth.TIMMainFrameSurfMaker
Program Setup:
Do/check this *before* you expect the app to run: (If you are doing this on Varro, our Fujitsu P1510, this is already done... location: /home/mobicomp/mobi_svn/...)
- Have a working version of the mobicomp jar, built with the 'ant linuxjmf' command.
- Have set up these hardcoded paths (I actually suggest making the dir structure according to these - easier) correctly before using the 'ant linuxjmf' command ;-):
1. line 36 of ETHImageTracker -> keyDir : absolute path of where your reference surf files (and nothing else!) live
2. line 38 of ETHImageTracker -> surfLibraryLocation : ... absolute path of your surf.ln file
3. line 41 of ETHCameraTracker -> jpgDir : absolute path of where taken snapshots are stored
4. line 40 of TIMMainFrameSurfMaker -> newDir : absolute path of where new snapshots with new surf files are stored
5. line 37 of ETHImageTracker -> htmlDir : absolute path of where html pages denoting exhibits are stored. Use this only if they are stored locally, else change to a URL. Also see line 214.
6. for now 'fingerprint.gif' and the 'dogruns.gif' have to be in the mobicomp dir of the mobicomp package you got out of subversion. They are curently stored in the src branch in the eth dir, so have to be moved.
These could eventually be stored in a properties file. TODO
"In-a-museum" Setup:
1. Make sure the Program Setup is complete.
2. Run the TIMMainFrameSurfMaker.
3. Take several pictures of each exhibit from different angles.
4. Make sure the app is generating surf files (as well as image files) in your newDir.
5. Copy the surf files only into your keyDir.
6. Run the app - try to recognize several objects, make sure nothing takes too long, etc
Things to note:
From my experiments with the code, simple exhibits make smaller surf files (fewer interesting points) and take less time to recognize. They also require a much lower threshold of interest points. (ETHImageTracker.threshold)
Visually complicated exhibits vice versa. So *I think* it is generally better to take fewer pics of complicated exhibits with many interest points as the algorithm seems to recognize them better anyway and takes too long if there are many comparisons to be made.
So depending on the overall nature of the exhibits in your museum scenario you might have to fine-tune the threshold. A low threshold works well for simple objects but makes overall accuracy go down, and a high threshold means good accuracy but also the fact that simple objects might never get recognized as they just do not have enough interest points on them to pass the 'recognition' vs. 'no match' threshold. Good luck fine tuning :-)
The recognizer works on all the image. This should be obvious, but it means that the background makes a big difference. If views of the same object from different angles have a different background (a table ledge ending and a floor of a different colour visible, etc..) make sure you have taken some snaps from these different angles so the object can be recognized even with different backgrounds.
