# Introducing LETOR 4.0 Datasets

arXiv: https://arxiv.org/abs/1306.2597 · HTML: https://mlsanity.com/paper/1306.2597/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2013.

## Built on by (most cited first, 30 of 121)

- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses](https://mlsanity.com/paper/2210.10634/) (2022)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [Neural Basis Models for Interpretability](https://mlsanity.com/paper/2205.14120/) (2022)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [SwitchTab: Switched Autoencoders Are Effective Tabular Learners](https://mlsanity.com/paper/2401.02013/) (2024)
- [B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2104.09791/) (2021)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise](https://mlsanity.com/paper/2408.09929/) (2024)
- [A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning](https://mlsanity.com/paper/2311.05877/) (2023)
- [Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence](https://mlsanity.com/paper/2202.12183/) (2022)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Toward Understanding Privileged Features Distillation in Learning-to-Rank](https://mlsanity.com/paper/2209.08754/) (2022)
- [Distributed Differential Privacy in Multi-Armed Bandits](https://mlsanity.com/paper/2206.05772/) (2022)
- [On Differentially Private Federated Linear Contextual Bandits](https://mlsanity.com/paper/2302.13945/) (2023)
- [Control Variates for Slate Off-Policy Evaluation](https://mlsanity.com/paper/2106.07914/) (2021)
- [Predictive Inference with Weak Supervision](https://mlsanity.com/paper/2201.08315/) (2022)
- [Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity](https://mlsanity.com/paper/2204.10872/) (2022)
- [Offline Model-Based Optimization by Learning to Rank](https://mlsanity.com/paper/2410.11502/) (2024)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)
- [Introducing the Expohedron for Efficient Pareto-optimal Fairness-Utility Amortizations in Repeated Rankings](https://mlsanity.com/paper/2202.03237/) (2022)
- [Towards Feature Selection for Ranking and Classification Exploiting Quantum Annealers](https://mlsanity.com/paper/2205.04346/) (2022)
- [PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.08872/) (2022)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)

## Abstract (arXiv)

LETOR is a package of benchmark data sets for research on LEarning TO Rank, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. This version, 4.0, was released in July 2009. Very different from previous versions (V3.0 is an update based on V2.0 and V2.0 is an update based on V1.0), LETOR4.0 is a totally new release. It uses the Gov2 web page collection (~25M pages) and two query sets from Million Query track of TREC 2007 and TREC 2008. We call the two query sets MQ2007 and MQ2008 for short. There are about 1700 queries in MQ2007 with labeled documents and about 800 queries in MQ2008 with labeled documents. If you have any questions or suggestions about the datasets, please kindly email us (letor@microsoft.com). Our goal is to make the dataset reliable and useful for the community.