Baselines on LETOR3.0

Algorithms with linear ranking function

TD2003 TD2004 NP2003 NP2004 HP2003 HP2004 OHSUMED Prediction files on test set Notes Experiments by
Regression here here here here here here here test scores Algorithm details Da Kuang
RankSVM here here here here here here here test scores Algorithm details Chaoliang Zhong
ListNet here here here here here here here test scores Algorithm details Da Kuang
AdaRank-MAP here here here here here here here test scores Algorithm details Chaoliang Zhong
AdaRank-NDCG here here here here here here here test scores Algorithm details Chaoliang Zhong
SVMMAP here here here here here here here not available Algorithm details Yisong Yue


Algorithms with nonlinear ranking function

TD2003 TD2004 NP2003 NP2004 HP2003 HP2004 OHSUMED Prediction files on test set Notes Experiments by
RankBoost here here here here here here here test scores Algorithm details Yong-Deok Kim
FRank here here here here here here here test scores Algorithm details Ming-Feng Tsai


Recently added algorithms (with linear ranking function)

Please note that the above experimental results are still primal, since the result of almost every algorithm can be further improved. For example, for regression, we can add regularization item to make it more robust; for RankSVM, we can run more steps of iteration so as to guarantee a better convergence of the optimization; for ListNet, we can also add regularization item to its loss function and make it more generalizable to the test set. Any updates about the above algorithms or new ranking algorithms are welcome. The following table lists the updated results of several algorithms (Regression and RankSVM) and a new algorithm SmoothRank.We would like to thank Dr. Olivier Chapelle and Prof. Thorsten Joachims for kindly contributing the results.
TD2003 TD2004 NP2003 NP2004 HP2003 HP2004 OHSUMED Prediction files on test set Notes Experiments by
Regression+L2 reg here here here here here here here Algorithm details Dr. Olivier Chapelle
RankSVM-Primal  here here here here here here here Algorithm details Dr. Olivier Chapelle
RankSVM-Struct here here here here here here here   Algorithm details Prof. Thorsten Joachims
SmoothRank here here here here here here here   Algorithm details Dr. Olivier Chapelle


Summary of all algorithms and datasets

Excel file

How to compare with the baselines?

We note that different setting of experiments may greatly affect the performance of a ranking algorithm. To make fair comparisons, we encourage everyone to follow these common settings while using LETOR; deviations from these defaults must be noted when reporting results.
  • All reported algorithms use the "QueryLevelNorm" version of the datasets (i.e. query level normalization for feature processing). You are encouraged to use the same version and should indicate if you use a different one.
  • The test set cannot be used in any manner to make decisions about the structure or parameters of the model.
  • The validation set can only be used for model selection (setting hyper-parameters and model structure), but cannot be used for learning. Most baselines released in LETOR website use MAP on the validation set for model selection; you are encouraged to use the same strategy and should indicate if you use a different one.
  • All reported results must use the provided evaluation utility. While using the evaluation script, please use the original dataset. The evaluation tool (Eval-Score-3.0.pl) sorts the documents with same ranking scores according to their input order. That is, it is sensitive to the document order in the input file.
  • Please explicitly show the function class of ranking models (e.g. linear model, two layer neural net, or decision trees) in your work.


Additional Notes

  • The prediction score files on test set can be viewed by any text editor such as notepad.
  • More algorithms will be added in future.
  • If you would be like to publish the results of your algorithm here, please let us know
2009 Microsoft Corporation. All rights reserved.  Terms of Use | Trademarks | Privacy Statement