# Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction

arXiv: https://arxiv.org/abs/2005.08480 · HTML: https://mlsanity.com/paper/2005.08480/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 10 of 10)

- [LiPO: Listwise Preference Optimization through Learning-to-Rank](https://mlsanity.com/paper/2402.01878/) (2024)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Towards Disentangling Relevance and Bias in Unbiased Learning to Rank](https://mlsanity.com/paper/2212.13937/) (2022)
- [COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation](https://mlsanity.com/paper/2210.15500/) (2022)
- [Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective](https://mlsanity.com/paper/2306.07528/) (2023)
- [Bilateral Self-unbiased Learning from Biased Implicit Feedback](https://mlsanity.com/paper/2207.12660/) (2022)
- [Biases in Scholarly Recommender Systems: Impact, Prevalence, and Mitigation](https://mlsanity.com/paper/2301.07483/) (2023)
- [Algorithmic Fairness: A Tolerance Perspective](https://mlsanity.com/paper/2405.09543/) (2024)
- [Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study](https://mlsanity.com/paper/2404.03707/) (2024)
- [Interaction-Data-guided Conditional Instrumental Variables for Debiasing Recommender Systems](https://mlsanity.com/paper/2408.09651/) (2024)

## Abstract (arXiv)

Recent advances in unbiased learning to rank (LTR) count on Inverse Propensity Scoring (IPS) to eliminate bias in implicit feedback. Though theoretically sound in correcting the bias introduced by treating clicked documents as relevant, IPS ignores the bias caused by (implicitly) treating non-clicked ones as irrelevant. In this work, we first rigorously prove that such use of click data leads to unnecessary pairwise comparisons between relevant documents, which prevent unbiased ranker optimization. Based on the proof, we derive a simple yet well justified new weighting scheme, called Propensity Ratio Scoring (PRS), which provides treatments on both clicks and non-clicks. Besides correcting the bias in clicks, PRS avoids relevant-relevant document comparisons in LTR training and enjoys a lower variability. Our extensive empirical evaluations confirm that PRS ensures a more effective use of click data and improved performance in both synthetic data from a set of LTR benchmarks, as well as in the real-world large-scale data from GMail search.