mlsanityData to 14 Oct 2026

Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset

arXiv 2404.02543 ↗First posted 3 Apr 2024cs.IR cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks, which are often biased by the ranker collecting the data. While theoretically justified and extensively tested in simulation, ULTR techniques lack empirical validation, especially on modern search engines. The Baidu-ULTR dataset released for the WSDM Cup 2023, collected from Baidu's search engine, offers a rare opportunity to assess the real-world performance of prominent ULTR techniques. Despite multiple submissions during the WSDM Cup 2023 and the subsequent NTCIR ULTRE-2 task, it remains unclear whether the observed improvements stem from applying ULTR or other learning techniques. In this work, we revisit and extend the available experiments on the Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques robustly improve click predictions but struggle to consistently improve ranking performance, especially considering the stark differences obtained by choice of ranking loss and query-document features. Our experiments reveal that gains in click prediction do not necessarily translate to enhanced ranking performance on expert relevance annotations, implying that conclusions strongly depend on how success is measured in this benchmark.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 26 of its 67 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

26 of 67 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Multi-Stage Document Ranking with BERTcited by 201
  3. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 157
  4. 2013Introducing LETOR 4.0 Datasetscited by 121
  5. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  6. 2019Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedbackcited by 79
  7. 2018Unbiased Learning to Rank with Unbiased Propensity Estimationcited by 61
  8. 2018Beyond the Selected Completely At Random Assumption for Learning from Positive and Unlabeled Datacited by 36
  9. 2018TF-Ranking: Scalable TensorFlow Library for Learning-to-Rankcited by 32
  10. 2018Estimating Position Bias without Intrusive Interventionscited by 32
  11. 2023Towards Better Web Search Performance: Pre-training, Fine-tuning and Learning to Rankcited by 17
  12. 2020Cascade Model-based Propensity Estimation for Counterfactual Learning to Rankcited by 15
14 more
  1. 2020Unifying Online and Counterfactual Learning to Rankcited by 15
  2. 2022A Large Scale Search Dataset for Unbiased Learning to Rankcited by 13
  3. 2023Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimizationcited by 11
  4. 2022Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rankcited by 10
  5. 2022Towards Disentangling Relevance and Bias in Unbiased Learning to Rankcited by 9
  6. 2021ULTRA: An Unbiased Learning To Rank Algorithm Toolboxcited by 6
  7. 2023An Offline Metric for the Debiasedness of Click Modelscited by 6
  8. 2023Multi-Feature Integration for Perception-Dependent Examination-Bias Estimationcited by 5
  9. 2023On the Impact of Outlier Bias on User Clickscited by 3
  10. 2023Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rankcited by 3
  11. 2023Feature-Enhanced Network with Hybrid Debiasing Strategies for Unbiased Learning to Rankcited by 2
  12. 2023Pretraining De-Biased Language Model with Large-scale Click Logs for Document Rankingcited by 2
  13. 2023CIR at the NTCIR-17 ULTRE-2 Taskcited by 2
  14. 2023Ensemble Ranking Model with Multiple Pretraining Strategies for Web Searchcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2025Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Modelscited by 4
  2. 2024Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Studycited by 1
  3. 2025A Control Function Framework for Mitigating Position Bias in Learning to Rank Systemscited by 1
  4. 2026Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenderscited by 1
  5. 2024Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rankcited by 0
  6. 2024Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Modelscited by 0
  7. 2025Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rankcited by 0
  8. 2025RewardRank: Optimizing True Learning-to-Rank Utilitycited by 0
  9. 2025CLAX: Fast and Flexible Neural Click Models in JAXcited by 0
  10. 2026MLPlatt: Simple Calibration Framework for Ranking Modelscited by 0