Computing Information Retrieval Performance Measures Efficiently in the Presence of Tied Scores

  • Marc Najork ,
  • Frank McSherry

30th European Conference on IR Research (ECIR) |

Published by Springer-Verlag

The Information Retrieval community uses a variety of performance measures to evaluate the effectiveness of scoring functions. In this paper, we show how to adapt six popular measures, precision, recall, F1, average precision, reciprocal rank, and normalized discounted cumulative gain, to cope with scoring functions that are likely to assign many tied scores to the results of a search. Tied scores impose only a partial ordering on the results, meaning that there are multiple possible orderings of the result set, each one performing differently. One approach to cope with ties would be to average the performance values across all possible result orderings; but unfortunately, generating result permutations requires super-exponential time. The approach presented in this paper computes precisely the same performance value as the approach of averaging over all permutations, but does so as efficiently as the original, tie-oblivious measures.

Publication Downloads

Tie-Aware Information-Retrieval Performance Measures

May 2, 2007

A collection of short programs to compute standard information-retrieval performance measures—Recall, Precision, F-measure, Mean Average Precision, Mean Reciprocal Rank, Normalized Discounted Cumulative Gain—in the presence of tied scores.