Learning to Search Web Pages with Query-Level Loss Functions

With the rapid development of information retrieval and Web search, ranking has become a new branch of supervised learning. Many existing machine learning technologies such as Support Vector Machines, Boosting, and Neural Networks have been applied to this new problem and achieved some success. However, since these algorithms are not proposed initially for information retrieval, their loss functions are not quite in accordance with widely-used evaluation criteria for information retrieval. Such criteria include mean average precision (MAP), mean precision at n (P@n), and normalized discounted cumulative gain (NDCG). While these criteria are averaged on the query-level, loss functions of the aforementioned machine learning algorithms are defined on the level of documents or document pairs. Therefore, it is not clear whether minimizing these loss functions can effectively improve the retrieval performance. To solve this problem, we propose to use query-level loss functions when learning the ranking function. In particular, we discuss what kind of properties a good query-level loss function should have, and propose a query-level loss function based on the cosine similarity between a ranking list and the corresponding ground truth with respect to a given query. In the meanwhile, we also discuss whether loss functions of existing ranking algorithms can be upgraded to the query level. Experiments on the datasets for TREC web track, OSHUMED, and a large collection of web pages from a commercial Web search engine all showed that our proposed query-level loss function can improve the search accuracy significantly, and it is non-trivial to upgrade traditional document-level loss functions to the query level.