Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Are click-through data adequate for learning web search rankings?

Zhicheng Dou, Ruihua Song, Xiaojie Yuan, and Ji-Rong Wen

Abstract

Learning-to-rank algorithms, which can automatically adapt ranking functions in web search, require a large volume of training data. A traditional way of generating training examples is to employ human experts to judge the relevance of documents. Unfortunately, it is difficult, time-consuming and costly. In this paper, we study the problem of exploiting click-through data for learning web search rankings that can be collected at much lower cost. We extract pairwise relevance preferences from a large-scale aggregated click-through dataset, compare these preferences with explicit human judgments, and use them as training examples to learn ranking functions. We find click-through data are useful and effective in learning ranking functions. A straightforward use of aggregated click-through data can outperform human judgments. We demonstrate that the strategies are only slightly affected by fraudulent clicks. We also reveal that the pairs which are very reliable, e.g., the pairs consisting of documents with large click frequency differences, are not sufficient for learning.

Details

Publication typeInproceedings
Published inCIKM '08: Proceeding of the 17th ACM conference on Information and knowledge management
URLhttp://doi.acm.org/10.1145/1458082.1458095
Pages73–82
ISBN978-1-59593-991-3
AddressNew York, NY, USA
PublisherACM
> Publications > Are click-through data adequate for learning web search rankings?