# Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank

arXiv: https://arxiv.org/abs/2407.19943 · HTML: https://mlsanity.com/paper/2407.19943/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 59 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Truly Proximal Policy Optimization](https://mlsanity.com/paper/1903.07940/) (2019)
- [Generalized Proximal Policy Optimization with Sample Reuse](https://mlsanity.com/paper/2111.00072/) (2021)
- [Trust Region-Guided Proximal Policy Optimization](https://mlsanity.com/paper/1901.10314/) (2019)
- [Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness](https://mlsanity.com/paper/2105.00855/) (2021)
- [Correcting for Selection Bias in Learning-to-rank Systems](https://mlsanity.com/paper/2001.11358/) (2020)
- [Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method](https://mlsanity.com/paper/2308.09861/) (2023)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Policy-Gradient Training of Fair and Unbiased Ranking Functions](https://mlsanity.com/paper/1911.08054/) (2019)
- [Unifying Online and Counterfactual Learning to Rank](https://mlsanity.com/paper/2012.04426/) (2020)
- [Safe Exploration for Optimizing Contextual Bandits](https://mlsanity.com/paper/2002.00467/) (2020)
- [StochasticRank: Global Optimization of Scale-Free Discrete Functions](https://mlsanity.com/paper/2003.02122/) (2020)
- [Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity](https://mlsanity.com/paper/2204.10872/) (2022)
- [Optimal Baseline Corrections for Off-Policy Contextual Bandits](https://mlsanity.com/paper/2405.05736/) (2024)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)
- [Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank](https://mlsanity.com/paper/2206.12204/) (2022)
- [Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank](https://mlsanity.com/paper/2102.05990/) (2021)
- [Learning from User Interactions with Rankings: A Unification of the Field](https://mlsanity.com/paper/2012.06576/) (2020)
- [A First Look at Selection Bias in Preference Elicitation for Recommendation](https://mlsanity.com/paper/2405.00554/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning](https://mlsanity.com/paper/2503.00897/) (2025)
- [Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank](https://mlsanity.com/paper/2409.09881/) (2024)
- [Optimizing Compound Retrieval Systems](https://mlsanity.com/paper/2504.12063/) (2025)
- [A Simpler Alternative to Variational Regularized Counterfactual Risk Minimization](https://mlsanity.com/paper/2409.09819/) (2024)
- [Counterfactual Inference under Thompson Sampling](https://mlsanity.com/paper/2504.08773/) (2025)
- [RewardRank: Optimizing True Learning-to-Rank Utility](https://mlsanity.com/paper/2508.14180/) (2025)
- [Safe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models](https://mlsanity.com/paper/2510.15429/) (2025)

## Abstract (arXiv)

Counterfactual learning to rank (CLTR) can be risky and, in various circumstances, can produce sub-optimal models that hurt performance when deployed. Safe CLTR was introduced to mitigate these risks when using inverse propensity scoring to correct for position bias. However, the existing safety measure for CLTR is not applicable to state-of-the-art CLTR methods, cannot handle trust bias, and relies on specific assumptions about user behavior. Our contributions are two-fold. First, we generalize the existing safe CLTR approach to make it applicable to state-of-the-art doubly robust CLTR and trust bias. Second, we propose a novel approach, proximal ranking policy optimization (PRPO), that provides safety in deployment without assumptions about user behavior. PRPO removes incentives for learning ranking behavior that is too dissimilar to a safe ranking model. Thereby, PRPO imposes a limit on how much learned models can degrade performance metrics, without relying on any specific user assumptions. Our experiments show that both our novel safe doubly robust method and PRPO provide higher performance than the existing safe inverse propensity scoring approach. However, in unexpected circumstances, the safe doubly robust approach can become unsafe and bring detrimental performance. In contrast, PRPO always maintains safety, even in maximally adversarial situations. By avoiding assumptions, PRPO is the first method with unconditional safety in deployment that translates to robust safety for real-world applications.