# Intervention Harvesting for Context-Dependent Examination-Bias Estimation

arXiv: https://arxiv.org/abs/1811.01802 · HTML: https://mlsanity.com/paper/1811.01802/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 14 of 14)

- [End-to-end Learning for Fair Ranking Systems](https://mlsanity.com/paper/2111.10723/) (2021)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022)
- [Offline Recommender System Evaluation under Unobserved Confounding](https://mlsanity.com/paper/2309.04222/) (2023)
- [Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank](https://mlsanity.com/paper/2309.15560/) (2023)
- [Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank](https://mlsanity.com/paper/2206.01702/) (2022)
- [Contextual bandits with concave rewards, and an application to fair ranking](https://mlsanity.com/paper/2210.09957/) (2022)
- [Biases in Scholarly Recommender Systems: Impact, Prevalence, and Mitigation](https://mlsanity.com/paper/2301.07483/) (2023)
- [Calibrating Explore-Exploit Trade-off for Fair Online Learning to Rank](https://mlsanity.com/paper/2111.00735/) (2021)
- [Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study](https://mlsanity.com/paper/2404.03707/) (2024)
- [A Control Function Framework for Mitigating Position Bias in Learning to Rank Systems](https://mlsanity.com/paper/2506.06989/) (2025)
- [Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank](https://mlsanity.com/paper/2506.20501/) (2025)
- [Learning to Rank with Variable Result Presentation Lengths](https://mlsanity.com/paper/2506.23319/) (2025)

## Abstract (arXiv)

Accurate estimates of examination bias are crucial for unbiased learning-to-rank from implicit feedback in search engines and recommender systems, since they enable the use of Inverse Propensity Score (IPS) weighting techniques to address selection biases and missing data. Unfortunately, existing examination-bias estimators are limited to the Position-Based Model (PBM), where the examination bias may only depend on the rank of the document. To overcome this limitation, we propose a Contextual Position-Based Model (CPBM) where the examination bias may also depend on a context vector describing the query and the user. Furthermore, we propose an effective estimator for the CPBM based on intervention harvesting. A key feature of the estimator is that it does not require disruptive interventions but merely exploits natural variation resulting from the use of multiple historic ranking functions. Real-world experiments on the ArXiv search engine and semi-synthetic experiments on the Yahoo Learning-To-Rank dataset demonstrate the superior effectiveness and robustness of the new approach.