Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
A few good topics: Experiments in topic set reduction for retrieval evaluation

John Guiver, Stefano Mizzaro, and Stephen Robertson

Abstract

We consider the issue of evaluating information retrieval systems on the basis of a limited number of topics. In contrast to statistically-based work on sample sizes, we hypothesise that some topics or topic sets are better than others at predicting true system effectiveness, and that with the right choice of topics, accurate predictions can be obtained from small topics sets. Using a variety of effectiveness metrics and measures of goodness of prediction, a study of a set of TREC and NTCIR results confirms this hypothesis, and provides evidence that the value of a topic set for this purpose does generalise.

Details

Publication typeArticle
Published inACM Transactions on Information Systems (TOIS) volume 27 issue 4
PublisherAssociation for Computing Machinery, Inc.
> Publications > A few good topics: Experiments in topic set reduction for retrieval evaluation