# Towards Disentangling Relevance and Bias in Unbiased Learning to Rank

arXiv: https://arxiv.org/abs/2212.13937 · HTML: https://mlsanity.com/paper/2212.13937/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 28 Dec 2022.

## Builds on (most cited first, 15 of 15)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Graph Enhanced Representation Learning for News Recommendation](https://mlsanity.com/paper/2003.14292/) (2020)
- [Improving Deep Learning For Airbnb Search](https://mlsanity.com/paper/2002.05515/) (2020)
- [Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank](https://mlsanity.com/paper/2005.11938/) (2020)
- [Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction](https://mlsanity.com/paper/2005.08480/) (2020)
- [It Is Different When Items Are Older: Debiasing Recommendations When Selection Bias and User Preferences Are Dynamic](https://mlsanity.com/paper/2111.12481/) (2021)
- [Unbiased Learning to Rank: Online or Offline?](https://mlsanity.com/paper/2004.13574/) (2020)
- [Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank](https://mlsanity.com/paper/2206.01702/) (2022)
- [A General Framework for Debiasing in CTR Prediction](https://mlsanity.com/paper/2112.02767/) (2021)

## Built on by (most cited first, 9 of 9)

- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank](https://mlsanity.com/paper/2309.15560/) (2023)
- [A Control Function Framework for Mitigating Position Bias in Learning to Rank Systems](https://mlsanity.com/paper/2506.06989/) (2025)
- [Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank](https://mlsanity.com/paper/2506.20501/) (2025)
- [Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models](https://mlsanity.com/paper/2409.12043/) (2024)
- [TSPRank: Bridging Pairwise and Listwise Methods with a Bilinear Travelling Salesman Model](https://mlsanity.com/paper/2411.12064/) (2024)
- [A Causal Information-Flow Framework for Unbiased Learning-to-Rank](https://mlsanity.com/paper/2601.05590/) (2026)
- [Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization](https://mlsanity.com/paper/2602.03570/) (2026)
- [SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking](https://mlsanity.com/paper/2607.20655/) (2026)

## Abstract (arXiv)

Unbiased learning to rank (ULTR) studies the problem of mitigating various biases from implicit user feedback data such as clicks, and has been receiving considerable attention recently. A popular ULTR approach for real-world applications uses a two-tower architecture, where click modeling is factorized into a relevance tower with regular input features, and a bias tower with bias-relevant inputs such as the position of a document. A successful factorization will allow the relevance tower to be exempt from biases. In this work, we identify a critical issue that existing ULTR methods ignored - the bias tower can be confounded with the relevance tower via the underlying true relevance. In particular, the positions were determined by the logging policy, i.e., the previous production model, which would possess relevance information. We give both theoretical analysis and empirical results to show the negative effects on relevance tower due to such a correlation. We then propose three methods to mitigate the negative confounding effects by better disentangling relevance and bias. Empirical results on both controlled public datasets and a large-scale industry dataset show the effectiveness of the proposed approaches.