# Pessimistic Off-Policy Optimization for Learning to Rank

arXiv: https://arxiv.org/abs/2206.02593 · HTML: https://mlsanity.com/paper/2206.02593/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 66 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism](https://mlsanity.com/paper/2103.12021/) (2021)
- [Bellman-consistent Pessimism for Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06926/) (2021)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Feedback Loop and Bias Amplification in Recommender Systems](https://mlsanity.com/paper/2007.13019/) (2020)
- [Cascading Bandits: Learning to Rank in the Cascade Model](https://mlsanity.com/paper/1502.02763/) (2015)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Balanced Policy Evaluation and Learning](https://mlsanity.com/paper/1705.07384/) (2017)
- [Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions](https://mlsanity.com/paper/2007.12986/) (2020)
- [Offline Evaluation of Ranking Policies with Click Models](https://mlsanity.com/paper/1804.10488/) (2018)
- [Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model](https://mlsanity.com/paper/2202.01562/) (2022)
- [DCM Bandits: Learning to Rank with Multiple Clicks](https://mlsanity.com/paper/1602.03146/) (2016)
- [Imitation-Regularized Offline Learning](https://mlsanity.com/paper/1901.04723/) (2019)
- [A/B Testing for Recommender Systems in a Two-sided Marketplace](https://mlsanity.com/paper/2106.00762/) (2021)
- [Personalized Web Search](https://mlsanity.com/paper/1502.01057/) (2015)

## Built on by (most cited first, 2 of 2)

- [Probabilistic Rank and Reward: A Scalable Model for Slate Recommendation](https://mlsanity.com/paper/2208.06263/) (2022)
- [Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective](https://mlsanity.com/paper/2306.07528/) (2023)

## Abstract (arXiv)

Off-policy learning is a framework for optimizing policies without deploying them, using data collected by another policy. In recommender systems, this is especially challenging due to the imbalance in logged data: some items are recommended and thus logged more frequently than others. This is further perpetuated when recommending a list of items, as the action space is combinatorial. To address this challenge, we study pessimistic off-policy optimization for learning to rank. The key idea is to compute lower confidence bounds on parameters of click models and then return the list with the highest pessimistic estimate of its value. This approach is computationally efficient, and we analyze it. We study its Bayesian and frequentist variants and overcome the limitation of unknown prior by incorporating empirical Bayes. To show the empirical effectiveness of our approach, we compare it to off-policy optimizers that use inverse propensity scores or neglect uncertainty. Our approach outperforms all baselines and is both robust and general.