Baselines on LETOR3.0
Algorithms with linear ranking function
Algorithms with nonlinear ranking function
Recently added algorithms (with linear ranking function)
Please note that the above experimental results are still primal, since the result
of almost every algorithm can be further improved. For example, for regression,
we can add regularization item to make it more robust; for RankSVM, we can run more
steps of iteration so as to guarantee a better convergence of the optimization;
for ListNet, we can also add regularization item to its loss function and make it
more generalizable to the test set. Any updates about the above algorithms or new
ranking algorithms are welcome. The following table lists the updated results of
several algorithms (Regression and RankSVM) and a new algorithm SmoothRank.We would
like to thank Dr. Olivier Chapelle and Prof. Thorsten Joachims for kindly contributing
the results.
Summary of all algorithms and datasets
Excel file
How to compare with the baselines?
We note that different setting of experiments may greatly affect the performance
of a ranking algorithm. To make fair comparisons, we encourage everyone to follow
these common settings while using LETOR; deviations from these defaults must be
noted when reporting results.
- All reported algorithms use the "QueryLevelNorm" version of the datasets (i.e.
query level normalization for feature processing). You are encouraged to use the
same version and should indicate if you use a different one.
- The test set cannot be used in any manner to make decisions about the structure
or parameters of the model.
- The validation set can only be used for model selection (setting hyper-parameters
and model structure), but cannot be used for learning. Most baselines released in
LETOR website use MAP on the validation set for model selection; you are encouraged
to use the same strategy and should indicate if you use a different one.
- All reported results must use the provided evaluation utility. While using the
evaluation script, please use the original dataset. The evaluation tool (Eval-Score-3.0.pl)
sorts the documents with same ranking scores according to their input order. That
is, it is sensitive to the document order in the input file.
- Please explicitly show the function class of ranking models (e.g. linear model,
two layer neural net, or decision trees) in your work.
Additional Notes
- The prediction score files on test set can be viewed by any text editor such
as notepad.
- More algorithms will be added in future.
- If you would be like to publish the results of your algorithm here, please let us know
