Object Instance Recognition

Have you ever wanted to quickly know more about a specific product, landmark, painting, or other object?  Instead of trying to find the right keyword to search the web, take a picture.  We're developing a research prototype called Lincoln that automatically recognizes objects within a picture, taken with a digital camera or cell phone, and finds relevant information on the web.

For identifying objects in images, we use a method that that recognizes specific instances of the object, such as a Coke can, or a painting of Mona Lisa. 


The system works by matching small patches of the image known as features.  These features contain pixel intensity information as well as their relative positions to other features.  For efficient retrieval and matching, we group the features into sets of three, known as triplets.   An example triplet is shown below.


Using an inverse look-up table, matching triplets are found between the user's query image and those within the training database.  To verify the correct match, the spatial relationship between the matching triplets is checked for consistency.  Finally, if a correct match is found, a website associated with the image is sent back to the user.  A technical report describing the approach can be found here.


Here are some examples of the system working:


User query image Matching image in database Webpage sent to user