# Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study

arXiv: https://arxiv.org/abs/2404.03707 · HTML: https://mlsanity.com/paper/2404.03707/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 65 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 21 of 21)

- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://mlsanity.com/paper/1610.08136/) (2016)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Correcting for Selection Bias in Learning-to-rank Systems](https://mlsanity.com/paper/2001.11358/) (2020)
- [Differentiable Unbiased Online Learning to Rank](https://mlsanity.com/paper/1809.08415/) (2018)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank](https://mlsanity.com/paper/2005.11938/) (2020)
- [Unifying Online and Counterfactual Learning to Rank](https://mlsanity.com/paper/2012.04426/) (2020)
- [Intervention Harvesting for Context-Dependent Examination-Bias Estimation](https://mlsanity.com/paper/1811.01802/) (2018)
- [A Large Scale Search Dataset for Unbiased Learning to Rank](https://mlsanity.com/paper/2207.03051/) (2022)
- [A Click Sequence Model for Web Search](https://mlsanity.com/paper/1805.03411/) (2018)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)
- [Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction](https://mlsanity.com/paper/2005.08480/) (2020)
- [Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank](https://mlsanity.com/paper/2206.12204/) (2022)
- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Efficient Exploration of Gradient Space for Online Learning to Rank](https://mlsanity.com/paper/1805.07317/) (2018)
- [Variance Reduction in Gradient Exploration for Online Learning to Rank](https://mlsanity.com/paper/1906.03766/) (2019)
- [Model-based Unbiased Learning to Rank](https://mlsanity.com/paper/2207.11785/) (2022)
- [Approximated Doubly Robust Search Relevance Estimation](https://mlsanity.com/paper/2208.07671/) (2022)

## Built on by (most cited first, 1 of 1)

- [Localization Boosting for Growth Markets: Mitigating Cross-Locale Behavioral Bias in Learning-to-Rank](https://mlsanity.com/paper/2605.11272/) (2026)

## Abstract (arXiv)

Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. While the CLTR models can be theoretically unbiased when the user behavior assumption is correct and the propensity estimation is accurate, their effectiveness is usually empirically evaluated via simulation-based experiments due to a lack of widely available, large-scale, real click logs. However, many previous simulation-based experiments are somewhat limited because they may have one or more of the following deficiencies: 1) using a weak production ranker to generate initial ranked lists, 2) relying on a simplified user simulation model to simulate user clicks, and 3) generating a fixed number of synthetic click logs. As a result, the robustness of CLTR models in complex and diverse situations is largely unknown and needs further investigation. To address this problem, in this paper, we aim to investigate the robustness of existing CLTR models in a reproducibility study with extensive simulation-based experiments that (1) use production rankers with different ranking performance, (2) leverage multiple user simulation models with different user behavior assumptions, and (3) generate different numbers of synthetic sessions for the training queries. We find that the IPS-DCM, DLA-PBM, and UPE models show better robustness under various simulation settings than other CLTR models. Moreover, existing CLTR models often fail to outperform naive click baselines when the production ranker is strong and the number of training sessions is limited, indicating a pressing need for new CLTR algorithms tailored to these conditions.