# Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank

arXiv: https://arxiv.org/abs/2309.15560 · HTML: https://mlsanity.com/paper/2309.15560/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 27 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Estimating Position Bias without Intrusive Interventions](https://mlsanity.com/paper/1812.05161/) (2018)
- [Identifiable Generative Models for Missing Not at Random Data Imputation](https://mlsanity.com/paper/2110.14708/) (2021)
- [Improving Deep Learning For Airbnb Search](https://mlsanity.com/paper/2002.05515/) (2020)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank](https://mlsanity.com/paper/2005.11938/) (2020)
- [Intervention Harvesting for Context-Dependent Examination-Bias Estimation](https://mlsanity.com/paper/1811.01802/) (2018)
- [A Large Scale Search Dataset for Unbiased Learning to Rank](https://mlsanity.com/paper/2207.03051/) (2022)
- [Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank](https://mlsanity.com/paper/2206.12204/) (2022)
- [Towards Disentangling Relevance and Bias in Unbiased Learning to Rank](https://mlsanity.com/paper/2212.13937/) (2022)
- [A Deep Recurrent Survival Model for Unbiased Ranking](https://mlsanity.com/paper/2004.14714/) (2020)
- [Intervention Efficient Algorithms for Approximate Learning of Causal Graphs](https://mlsanity.com/paper/2012.13976/) (2020)
- [Multi-Feature Integration for Perception-Dependent Examination-Bias Estimation](https://mlsanity.com/paper/2302.13756/) (2023)
- [Counterfactual Learning to Rank using Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2007.09798/) (2020)
- [On the Impact of Outlier Bias on User Clicks](https://mlsanity.com/paper/2305.00857/) (2023)
- [Eliminating Search Intent Bias in Learning to Rank](https://mlsanity.com/paper/2002.03203/) (2020)
- [Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank](https://mlsanity.com/paper/2206.01702/) (2022)

## Built on by (most cited first, 3 of 3)

- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank](https://mlsanity.com/paper/2506.20501/) (2025)
- [Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models](https://mlsanity.com/paper/2409.12043/) (2024)

## Abstract (arXiv)

Unbiased Learning to Rank (ULTR) aims to train unbiased ranking models from biased click logs, by explicitly modeling a generation process for user behavior and fitting click data based on examination hypothesis. Previous research found empirically that the true latent relevance is mostly recoverable through click fitting. However, we demonstrate that this is not always achievable, resulting in a significant reduction in ranking performance. This research investigates the conditions under which relevance can be recovered from click data in the first principle. We initially characterize a ranking model as identifiable if it can recover the true relevance up to a scaling transformation, a criterion sufficient for the pairwise ranking objective. Subsequently, we investigate an equivalent condition for identifiability, articulated as a graph connectivity test problem: the recovery of relevance is feasible if and only if the identifiability graph (IG), derived from the underlying structure of the dataset, is connected. The presence of a disconnected IG may lead to degenerate cases and suboptimal ranking performance. To tackle this challenge, we introduce two methods, namely node intervention and node merging, designed to modify the dataset and restore the connectivity of the IG. Empirical results derived from a simulated dataset and two real-world LTR benchmark datasets not only validate our proposed theory but also demonstrate the effectiveness of our methods in alleviating data bias when the relevance model is unidentifiable.