## Baselines on LETOR3.0

### Algorithms with linear ranking function

TD2003 | TD2004 | NP2003 | NP2004 | HP2003 | HP2004 | OHSUMED | Prediction files on test set | Notes | Experiments by | |
---|---|---|---|---|---|---|---|---|---|---|

Regression | here | here | here | here | here | here | here | test scores | Algorithm details | Da Kuang |

RankSVM | here | here | here | here | here | here | here | test scores | Algorithm details | Chaoliang Zhong |

ListNet | here | here | here | here | here | here | here | test scores | Algorithm details | Da Kuang |

AdaRank-MAP | here | here | here | here | here | here | here | test scores | Algorithm details | Chaoliang Zhong |

AdaRank-NDCG | here | here | here | here | here | here | here | test scores | Algorithm details | Chaoliang Zhong |

SVMMAP | here | here | here | here | here | here | here | not available | Algorithm details | Yisong Yue |

### Algorithms with nonlinear ranking function

TD2003 | TD2004 | NP2003 | NP2004 | HP2003 | HP2004 | OHSUMED | Prediction files on test set | Notes | Experiments by | |
---|---|---|---|---|---|---|---|---|---|---|

RankBoost | here | here | here | here | here | here | here | test scores | Algorithm details | Yong-Deok Kim |

FRank | here | here | here | here | here | here | here | test scores | Algorithm details | Ming-Feng Tsai |

### Recently added algorithms (with linear ranking function)

Please note that the above experimental results are still primal, since the result of almost every algorithm can be further improved. For example, for regression, we can add regularization item to make it more robust; for RankSVM, we can run more steps of iteration so as to guarantee a better convergence of the optimization; for ListNet, we can also add regularization item to its loss function and make it more generalizable to the test set. Any updates about the above algorithms or new ranking algorithms are welcome. The following table lists the updated results of several algorithms (Regression and RankSVM) and a new algorithm SmoothRank.We would like to thank Dr. Olivier Chapelle and Prof. Thorsten Joachims for kindly contributing the results.TD2003 | TD2004 | NP2003 | NP2004 | HP2003 | HP2004 | OHSUMED | Prediction files on test set | Notes | Experiments by | |
---|---|---|---|---|---|---|---|---|---|---|

Regression+L2 reg | here | here | here | here | here | here | here | Algorithm details | Dr. Olivier Chapelle | |

RankSVM-Primal | here | here | here | here | here | here | here | Algorithm details | Dr. Olivier Chapelle | |

RankSVM-Struct | here | here | here | here | here | here | here | Algorithm details | Prof. Thorsten Joachims | |

SmoothRank | here | here | here | here | here | here | here | Algorithm details | Dr. Olivier Chapelle |

### Summary of all algorithms and datasets

Excel file### How to compare with the baselines?

We note that different setting of experiments may greatly affect the performance of a ranking algorithm. To make fair comparisons, we encourage everyone to follow these common settings while using LETOR; deviations from these defaults must be noted when reporting results.- All reported algorithms use the "QueryLevelNorm" version of the datasets (i.e. query level normalization for feature processing). You are encouraged to use the same version and should indicate if you use a different one.
- The test set cannot be used in any manner to make decisions about the structure or parameters of the model.
- The validation set can only be used for model selection (setting hyper-parameters and model structure), but cannot be used for learning. Most baselines released in LETOR website use MAP on the validation set for model selection; you are encouraged to use the same strategy and should indicate if you use a different one.
- All reported results must use the provided evaluation utility. While using the evaluation script, please use the original dataset. The evaluation tool (Eval-Score-3.0.pl) sorts the documents with same ranking scores according to their input order. That is, it is sensitive to the document order in the input file.
- Please explicitly show the function class of ranking models (e.g. linear model, two layer neural net, or decision trees) in your work.

### Additional Notes

- The prediction score files on test set can be viewed by any text editor such as notepad.
- More algorithms will be added in future.
- If you would be like to publish the results of your algorithm here, please let us know