LETOR: Learning to Rank for Information Retrieval
LETOR is a package of benchmark data sets for research on LEarning TO Rank released by Microsoft Research Asia, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. Version 4.0 was released in July 2009.
LETOR 4.0 was released in July 2009.
- Two large scale query sets were used, with thousands of queries;
- Datasets for four kinds of ranking settings were provided: supervised ranking, semi-supervised ranking, rank aggregation, and listwise ranking;
- Low level features were included for investigation;
- Several baselines were included.
LETOR 3.0 baselines were updated in June 2009.
LETOR 3.0 was released in Dec, 2008.
- Add four new datasets: homepage finding 2003, homepage finding 2004, named page finding 2003 and named page finding 2004. Plus the three datasets (OHSUMED, topic distillation 2003 and topic distillation 3004) in LETOR2.0, there are seven datasets in LETOR3.0.
- New document sampling strategy for each query; and so the three datasets in LETOR3.0 are different from those in LETOR2.0;
- New low level features for learning;
- Meta data is provided for better investigation of ranking features;
- More baselines.
LETOR 2.0 was released in December, 2007.
- More baseline results on the LETOR dataset, including ListNet, AdaRank, FRank and MHR.
- Updated evaluation tools (Eval-Rank.pl and Eval-ttest.pl).
- A channel to accept more baselines from other researchers.
- A discussion board for researchers to exchange their ideas on learning to rank and the LETOR dataset.
- A hub for researcher/research groups, papers and resources on learning to rank.
