Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval

  • James Philbin ,
  • Josef Sivic ,
  • Michael Isard ,
  • Andrew Zisserman ,
  • Ondřej Chum

IEEE International Conference on Computer Vision |

Given a query image of an object, our objective is to retrieve
all instances of that object in a large (1M+) image
database. We adopt the bag-of-visual-words architecture
which has proven successful in achieving high precision at
low recall. Unfortunately, feature detection and quantization
are noisy processes and this can result in variation in
the particular visual words that appear in different images
of the same object, leading to missed results.
In the text retrieval literature a standard method for improving
performance is query expansion. A number of the
highly ranked documents from the original query are reissued
as a new query. In this way, additional relevant terms
can be added to the query. This is a form of blind relevance
feedback and it can fail if ‘outlier’ (false positive)
documents are included in the reissued query.
In this paper we bring query expansion into the visual
domain via two novel contributions. Firstly, strong spatial
constraints between the query image and each result allow
us to accurately verify each return, suppressing the false
positives which typically ruin text-based query expansion.
Secondly, the verified images can be used to learn a latent
feature model to enable the controlled construction of expanded
queries.
We illustrate these ideas on the 5000 annotated image
Oxford building database together with more than 1M
Flickr images. We show that the precision is substantially
boosted, achieving total recall in many cases.