Project overview

Traditional IR evaluation is based on implicit assumptions about the user's interaction with the search system: It is assumed that the user is presented with a ranked list of results and examines the returned documents one after the other in the order they are listed. However, such a model is obsolete in the case of the Web and structured document retrieval, where browsing is an integral part of the user's search strategy.

The goal of this project is to better understand users' post-query browsing behaviour, and to develop appropriate evaluation measures that incorporate richer models of user interaction, taking into account browsing.

Our approach to capture retrieval effectiveness for query and navigation based search is to introduce a measure of retrieval effectiveness that comprises a probabilistic model of the users’ post-query navigation. Building on this, our measure of effort-precision and gain-recall (ep/gr) allows to explicitly model both the benefits and costs of discovering additional relevant information while browsing from a result in the ranked list. Our measure of Structural Relevance (SR) builds on a model of tree retrieval and a Markovian model of user navigation which eliminates the need for the computation of an ideal ranking.

News

• Our paper on SR, presented at CIKM 2008, won Best paper runner up prize.

Project team and collaborators

• Sadek Ali (University of Toronto)
• Mariano Consens (University of Toronto)
• Gabriella Kazai (Microsoft Research)
• Mounia Lalmas (University of Glasgow)
• Natasa Milic-Frayling (Microsoft Research)
• Benjamin Piwowarski (University of Glasgow)
• Stephen Robertson (Microsoft Research)
• Andrew Trotman (University of Otago)