# Correcting for Selection Bias in Learning-to-rank Systems

arXiv: https://arxiv.org/abs/2001.11358 · HTML: https://mlsanity.com/paper/2001.11358/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2020.

## Built on by (most cited first, 30 of 35)

- [AutoDebias: Learning to Debias for Recommendation](https://mlsanity.com/paper/2105.04170/) (2021)
- [CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation](https://mlsanity.com/paper/2403.06447/) (2024)
- [TFROM: A Two-sided Fairness-Aware Recommendation Model for Both Customers and Providers](https://mlsanity.com/paper/2104.09024/) (2021)
- [SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios](https://mlsanity.com/paper/2110.06475/) (2021)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Deconfounded Causal Collaborative Filtering](https://mlsanity.com/paper/2110.07122/) (2021)
- [Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobao](https://mlsanity.com/paper/2306.03527/) (2023)
- [Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation](https://mlsanity.com/paper/2310.16738/) (2023)
- [RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems](https://mlsanity.com/paper/2201.04399/) (2022)
- [Query Understanding in the Age of Large Language Models](https://mlsanity.com/paper/2306.16004/) (2023)
- [Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank](https://mlsanity.com/paper/2407.19943/) (2024)
- [PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation](https://mlsanity.com/paper/2601.18777/) (2026)
- [A Transformer-Based Substitute Recommendation Model Incorporating Weakly Supervised Customer Behavior Data](https://mlsanity.com/paper/2211.02533/) (2022)
- [Code Search Debiasing:Improve Search Results beyond Overall Ranking Performance](https://mlsanity.com/paper/2311.14901/) (2023)
- [Bilateral Self-unbiased Learning from Biased Implicit Feedback](https://mlsanity.com/paper/2207.12660/) (2022)
- [An Adaptive Boosting Technique to Mitigate Popularity Bias in Recommender System](https://mlsanity.com/paper/2109.05677/) (2021)
- [Augment & Valuate : A Data Enhancement Pipeline for Data-Centric AI](https://mlsanity.com/paper/2112.03837/) (2021)
- [Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank](https://mlsanity.com/paper/2206.01702/) (2022)
- [Measurement and applications of position bias in a marketplace search engine](https://mlsanity.com/paper/2206.11720/) (2022)
- [Biases in Scholarly Recommender Systems: Impact, Prevalence, and Mitigation](https://mlsanity.com/paper/2301.07483/) (2023)
- [Reducing Popularity Influence by Addressing Position Bias](https://mlsanity.com/paper/2412.08780/) (2024)
- [REST: Debiased Social Recommendation via Reconstructing Exposure Strategies](https://mlsanity.com/paper/2201.04952/) (2022)
- [Mitigating Frequency Bias in Next-Basket Recommendation via Deconfounders](https://mlsanity.com/paper/2211.09072/) (2022)
- [Recent Advances in the Foundations and Applications of Unbiased Learning to Rank](https://mlsanity.com/paper/2305.02914/) (2023)
- [Addressing Selection Bias in Computerized Adaptive Testing: A User-Wise Aggregate Influence Function Approach](https://mlsanity.com/paper/2308.11912/) (2023)
- [Position bias in features](https://mlsanity.com/paper/2402.02626/) (2024)
- [Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study](https://mlsanity.com/paper/2404.03707/) (2024)
- [The Importance of Causality in Decision Making: A Perspective on Recommender Systems](https://mlsanity.com/paper/2410.01822/) (2024)
- [A Control Function Framework for Mitigating Position Bias in Learning to Rank Systems](https://mlsanity.com/paper/2506.06989/) (2025)
- [Fairness Aware Reward Optimization](https://mlsanity.com/paper/2602.07799/) (2026)

## Abstract (arXiv)

Click data collected by modern recommendation systems are an important source of observational data that can be utilized to train learning-to-rank (LTR) systems. However, these data suffer from a number of biases that can result in poor performance for LTR systems. Recent methods for bias correction in such systems mostly focus on position bias, the fact that higher ranked results (e.g., top search engine results) are more likely to be clicked even if they are not the most relevant results given a user's query. Less attention has been paid to correcting for selection bias, which occurs because clicked documents are reflective of what documents have been shown to the user in the first place. Here, we propose new counterfactual approaches which adapt Heckman's two-stage method and accounts for selection and position bias in LTR systems. Our empirical evaluation shows that our proposed methods are much more robust to noise and have better accuracy compared to existing unbiased LTR algorithms, especially when there is moderate to no position bias.