# Estimating Position Bias without Intrusive Interventions

arXiv: https://arxiv.org/abs/1812.05161 · HTML: https://mlsanity.com/paper/1812.05161/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2018.

## Built on by (most cited first, 30 of 32)

- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [End-to-end Learning for Fair Ranking Systems](https://mlsanity.com/paper/2111.10723/) (2021)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [On component interactions in two-stage recommender systems](https://mlsanity.com/paper/2106.14979/) (2021)
- [Deconfounded Causal Collaborative Filtering](https://mlsanity.com/paper/2110.07122/) (2021)
- [Recent Developments in Recommender Systems: A Survey](https://mlsanity.com/paper/2306.12680/) (2023)
- [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022)
- [Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment](https://mlsanity.com/paper/2210.13005/) (2022)
- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Towards Fair Recommendation in Two-Sided Platforms](https://mlsanity.com/paper/2201.01180/) (2021)
- [Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank](https://mlsanity.com/paper/2102.05990/) (2021)
- [ULTRA: An Unbiased Learning To Rank Algorithm Toolbox](https://mlsanity.com/paper/2108.05073/) (2021)
- [Simpson's Paradox in Recommender Fairness: Reconciling differences between per-user and aggregated evaluations](https://mlsanity.com/paper/2210.07755/) (2022)
- [Click-Conversion Multi-Task Model with Position Bias Mitigation for Sponsored Search in eCommerce](https://mlsanity.com/paper/2307.16060/) (2023)
- [Code Search Debiasing:Improve Search Results beyond Overall Ranking Performance](https://mlsanity.com/paper/2311.14901/) (2023)
- [Uncertainty-Aware Instance Reweighting for Off-Policy Learning](https://mlsanity.com/paper/2303.06389/) (2023)
- [Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank](https://mlsanity.com/paper/2309.15560/) (2023)
- [Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank](https://mlsanity.com/paper/2206.01702/) (2022)
- [Measurement and applications of position bias in a marketplace search engine](https://mlsanity.com/paper/2206.11720/) (2022)
- [Biases in Scholarly Recommender Systems: Impact, Prevalence, and Mitigation](https://mlsanity.com/paper/2301.07483/) (2023)
- [Calibrating Explore-Exploit Trade-off for Fair Online Learning to Rank](https://mlsanity.com/paper/2111.00735/) (2021)
- [Low-variance estimation in the Plackett-Luce model via quasi-Monte Carlo sampling](https://mlsanity.com/paper/2205.06024/) (2022)
- [Off-policy evaluation for learning-to-rank via interpolating the item-position model and the position-based model](https://mlsanity.com/paper/2210.09512/) (2022)
- [Position bias in features](https://mlsanity.com/paper/2402.02626/) (2024)
- [The Importance of Causality in Decision Making: A Perspective on Recommender Systems](https://mlsanity.com/paper/2410.01822/) (2024)
- [A Control Function Framework for Mitigating Position Bias in Learning to Rank Systems](https://mlsanity.com/paper/2506.06989/) (2025)
- [Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank](https://mlsanity.com/paper/2506.20501/) (2025)
- [Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines](https://mlsanity.com/paper/2106.07719/) (2021)
- [Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank](https://mlsanity.com/paper/2408.09817/) (2024)
- [Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation](https://mlsanity.com/paper/2602.07298/) (2026)

## Abstract (arXiv)

Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal. While it was recently shown how counterfactual learning-to-rank (LTR) approaches citeJoachims/etal/17a can provably overcome presentation bias when observation propensities are known, it remains to show how to effectively estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model. Second, we propose a new extremum estimator that makes effective use of this data. In an empirical evaluation, we find that the new estimator provides superior propensity estimates in two real-world systems -- Arxiv Full-text Search and Google Drive Search. Beyond these two points, we find that the method is robust to a wide range of settings in simulation studies.