SIGIR 2008 Workshop: Learning to Rank for Information Retrieval


The task of "learning to rank" has emerged as an active and growing area of research both in information retrieval and machine learning. The goal is to design and apply methods to automatically learn a function from training data, such that the function can sort objects (e.g., documents) according to their degrees of relevance, preference, or importance as defined in a specific application.

The relevance of this task for IR is without question, because many IR problems are by nature ranking problems. Improved algorithms for learning ranking functions promise improved retrieval quality and less of a need for manual parameter adaptation. In this way, many IR technologies can be potentially enhanced by using learning to rank techniques.

A workshop entitled Learning to Rank for Information Retrieval will be organized at SIGIR 2008, following a very successful workshop on the same topic at SIGIR 2007 (website:  report: ). The main purpose of this workshop is to bring together IR researchers and ML researchers working on or interested in the technologies, and let them to share their latest research results, to express their opinions on the related issues, and to discuss future directions. 

Topics of Interests

We solicit submissions on any aspect of learning to rank for information retrieval. Particular areas of interest include, but are not limited to:


l  Models, features, and algorithms of learning to rank

l  Evaluation methods for learning to rank

l  Data creation methods for learning to rank

l  Applications of learning to rank methods to information retrieval

l  Comparison between traditional approaches and learning approaches to ranking

l  Theoretical analyses on learning to rank

l  Empirical comparison between learning to rank methods


Shared Benchmark Data

Several shared data sets have been released from Microsoft Research Asia ( The data sets, created based on OHSUMED and TREC data, contain features and relevance judgments for training and evaluation of learning to rank methods. It is encouraged to use the data sets to conduct experiments in the submissions to the workshop.

Paper Submission

Papers should be submitted electronically via the submission site ( Submitted papers should be in the ACM Conference style, see the ACM template page, and may not exceed 8 pages. All submissions will be reviewed by at least three members of the program committee. The review is double-blind; authors should conceal their identity where it is practical to do so. All accepted papers will be published in the proceedings of the workshop. The proceedings will be printed and made available at the workshop. At least one author should register and attend the workshop.


9:00~10:00    Keynote I: On the Optimisation of Evaluation Metrics (by Stephen Robertson)

10:00~10:30    Coffee break

10:30~12:30    Session 1: Learning to Rank Algorithms - I

    1) SortNet: Learning To Rank By a Neural-Based Sorting Algorithm (Leonardo Rigutini, Tiziano Papini, Marco Maggini, Franco Scarselli)

    2) Query-Level Learning to Rank Using Isotonic Regression (Zhaohui Zheng, Hongyuan Zha, Gordon Sun)

    3) A Meta-Learning Approach for Robust Rank Learning (Vitor R. Carvalho, Jonathan L. Elsas, William W. Cohen, Jaime G. Carbonell)
    4) A Decision Theoretic Framework for Ranking using Implicit Feedback (Onno Zoeter, Michael Taylor, Ed Snelson, John Guiver, Nick Craswell, Martin Szummer)

12:30~13:30    Lunch

13:30~14:30    Keynote II: A Structured Learning Framework for Learning to Rank in Web Search (by Hongyuan Zha)

14:30~15:30    Session 2: Learning to Rank Algorithms - II

    1) A Framework for Unsupervised Rank Aggregation (Alexandre Klementiev, Dan Roth, Kevin Small)

    2) Machine Learned Sentence Selection Strategies for Query-Biased Summarization (Donald Metzler, Tapas Kanungo)

15:30~16:00    Coffee break

16:00~17:00    Session 3: Benchmark Dataset for Learning to Rank

    1) Selection Bias in the LETOR Datasets (Tom Minka, Stephen Robertson)

    2) How to Make LETOR More Useful and Reliable (Tao Qin, Tie-Yan Liu, Jun Xu, Hang Li)

17:00~18:00    Session 4: Opinions on Learning to Rank


Workshop proceedings can be downloaded here.

Organizers / Co-chairs

Hang Li, Microsoft Research Asia

Tie-Yan Liu, Microsoft Research Asia

ChengXiang Zhai, Univ. of Illinois at Urbana-Champaign

Program Committee

Alekh Agarwal, University of California at Berkeley
Djoerd Hiemstra, University of Twente
Donald Metzler, University Massachusetts
Einat Minkov, Carnegie Mellon University
Filip Radlinski, Cornell University
Guirong Xue, Shanghai Jiao-Tong University
Guy Lebanon, Prudue University
Hongyuan Zha, Georgia Tech
Hsin-Hsi Chen, National University of Taiwan
Irina Matveeva, University of Chicago
Javed Aslam, Northeastern University
John Guiver, Microsoft Research Cambridge
Jun Xu, Microsoft Research Asia
Kai Yu, NEC Research Institute
Michael Taylor, Microsoft Research Cambridge
Olivier Chapelle, Yahoo Research
Ping Li, Cornell University
Quoc Le, Australian National University
Ralph Herbrich, Microsoft Research Cambridge
Ravi Kumar, Yahoo Research
Tao Qin, Tsinghua University
Yisong Yue, Cornell university
Zhaohui Zheng, Yahoo Research
Soumen Chakrabarti, Indian Institute of Technology

Important Dates

  • Paper Submission Due Extended: May  23 (23:59, Hawaii time)
  • Author Notification Date: June 11
  • Camera Ready: June 20
  • Workshop: July 24

Contact Us

Tie-Yan Liu, Microsoft Research Asia

tyliu [at] microsoft [dot] com


back to top