# Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset

arXiv: https://arxiv.org/abs/2404.02543 · HTML: https://mlsanity.com/paper/2404.02543/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback](https://mlsanity.com/paper/1909.03601/) (2019)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Beyond the Selected Completely At Random Assumption for Learning from Positive and Unlabeled Data](https://mlsanity.com/paper/1809.03207/) (2018)
- [TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank](https://mlsanity.com/paper/1812.00073/) (2018)
- [Estimating Position Bias without Intrusive Interventions](https://mlsanity.com/paper/1812.05161/) (2018)
- [Towards Better Web Search Performance: Pre-training, Fine-tuning and Learning to Rank](https://mlsanity.com/paper/2303.04710/) (2023)
- [Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank](https://mlsanity.com/paper/2005.11938/) (2020)
- [Unifying Online and Counterfactual Learning to Rank](https://mlsanity.com/paper/2012.04426/) (2020)
- [A Large Scale Search Dataset for Unbiased Learning to Rank](https://mlsanity.com/paper/2207.03051/) (2022)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)
- [Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank](https://mlsanity.com/paper/2206.12204/) (2022)
- [Towards Disentangling Relevance and Bias in Unbiased Learning to Rank](https://mlsanity.com/paper/2212.13937/) (2022)
- [ULTRA: An Unbiased Learning To Rank Algorithm Toolbox](https://mlsanity.com/paper/2108.05073/) (2021)
- [An Offline Metric for the Debiasedness of Click Models](https://mlsanity.com/paper/2304.09560/) (2023)
- [Multi-Feature Integration for Perception-Dependent Examination-Bias Estimation](https://mlsanity.com/paper/2302.13756/) (2023)
- [On the Impact of Outlier Bias on User Clicks](https://mlsanity.com/paper/2305.00857/) (2023)
- [Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank](https://mlsanity.com/paper/2309.15560/) (2023)
- [Feature-Enhanced Network with Hybrid Debiasing Strategies for Unbiased Learning to Rank](https://mlsanity.com/paper/2302.07530/) (2023)
- [Pretraining De-Biased Language Model with Large-scale Click Logs for Document Ranking](https://mlsanity.com/paper/2302.13498/) (2023)
- [CIR at the NTCIR-17 ULTRE-2 Task](https://mlsanity.com/paper/2310.11852/) (2023)
- [Ensemble Ranking Model with Multiple Pretraining Strategies for Web Search](https://mlsanity.com/paper/2302.09340/) (2023)

## Built on by (most cited first, 10 of 10)

- [Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models](https://mlsanity.com/paper/2505.07105/) (2025)
- [Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study](https://mlsanity.com/paper/2404.03707/) (2024)
- [A Control Function Framework for Mitigating Position Bias in Learning to Rank Systems](https://mlsanity.com/paper/2506.06989/) (2025)
- [Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders](https://mlsanity.com/paper/2607.24869/) (2026)
- [Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank](https://mlsanity.com/paper/2408.09817/) (2024)
- [Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models](https://mlsanity.com/paper/2409.12043/) (2024)
- [Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank](https://mlsanity.com/paper/2507.20753/) (2025)
- [RewardRank: Optimizing True Learning-to-Rank Utility](https://mlsanity.com/paper/2508.14180/) (2025)
- [CLAX: Fast and Flexible Neural Click Models in JAX](https://mlsanity.com/paper/2511.03620/) (2025)
- [MLPlatt: Simple Calibration Framework for Ranking Models](https://mlsanity.com/paper/2601.08345/) (2026)

## Abstract (arXiv)

Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks, which are often biased by the ranker collecting the data. While theoretically justified and extensively tested in simulation, ULTR techniques lack empirical validation, especially on modern search engines. The Baidu-ULTR dataset released for the WSDM Cup 2023, collected from Baidu's search engine, offers a rare opportunity to assess the real-world performance of prominent ULTR techniques. Despite multiple submissions during the WSDM Cup 2023 and the subsequent NTCIR ULTRE-2 task, it remains unclear whether the observed improvements stem from applying ULTR or other learning techniques. In this work, we revisit and extend the available experiments on the Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques robustly improve click predictions but struggle to consistently improve ranking performance, especially considering the stark differences obtained by choice of ranking loss and query-document features. Our experiments reveal that gains in click prediction do not necessarily translate to enhanced ranking performance on expert relevance annotations, implying that conclusions strongly depend on how success is measured in this benchmark.