# Optimal Baseline Corrections for Off-Policy Contextual Bandits

arXiv: https://arxiv.org/abs/2405.05736 · HTML: https://mlsanity.com/paper/2405.05736/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation](https://mlsanity.com/paper/2008.07146/) (2020)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Carousel Personalization in Music Streaming Apps with Contextual Bandits](https://mlsanity.com/paper/2009.06546/) (2020)
- [BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signals](https://mlsanity.com/paper/2008.12504/) (2020)
- [Long-Term Value of Exploration: Measurements, Findings and Algorithms](https://mlsanity.com/paper/2305.07764/) (2023)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)
- [Practical Bandits: An Industry Perspective](https://mlsanity.com/paper/2302.01223/) (2023)
- [Online Matching: A Real-time Bandit System for Large-scale Recommendations](https://mlsanity.com/paper/2307.15893/) (2023)
- [Variance Reduction in Ratio Metrics for Efficient Online Experiments](https://mlsanity.com/paper/2401.04062/) (2024)
- [Let's Get It Started: Fostering the Discoverability of New Releases on Deezer](https://mlsanity.com/paper/2401.02827/) (2024)

## Built on by (most cited first, 13 of 13)

- [A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning](https://mlsanity.com/paper/2503.00897/) (2025)
- [Delta-rm OPE: Off-Policy Estimation with Pairs of Policies](https://mlsanity.com/paper/2405.10024/) (2024)
- [Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank](https://mlsanity.com/paper/2407.19943/) (2024)
- [Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation](https://mlsanity.com/paper/2602.14914/) (2026)
- [Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank](https://mlsanity.com/paper/2409.09881/) (2024)
- [Unifying On- and Off-Policy Variance Reduction Methods](https://mlsanity.com/paper/2603.08370/) (2026)
- [A Simpler Alternative to Variational Regularized Counterfactual Risk Minimization](https://mlsanity.com/paper/2409.09819/) (2024)
- [Counterfactual Inference under Thompson Sampling](https://mlsanity.com/paper/2504.08773/) (2025)
- [Exploitation Over Exploration: Unmasking the Bias in Linear Bandit Recommender Offline Evaluation](https://mlsanity.com/paper/2507.18756/) (2025)
- [Meta Off-Policy Estimation](https://mlsanity.com/paper/2508.07914/) (2025)
- [Safe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models](https://mlsanity.com/paper/2510.15429/) (2025)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)
- [Exposure-Based Reinforcement Learning to Rank](https://mlsanity.com/paper/2607.18689/) (2026)

## Abstract (arXiv)

The off-policy learning paradigm allows for recommender systems and general ranking applications to be framed as decision-making problems, where we aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. With unbiasedness comes potentially high variance, and prevalent methods exist to reduce estimation variance. These methods typically make use of control variates, either additive (i.e., baseline corrections or doubly robust methods) or multiplicative (i.e., self-normalisation). Our work unifies these approaches by proposing a single framework built on their equivalence in learning scenarios. The foundation of our framework is the derivation of an equivalent baseline correction for all of the existing control variates. Consequently, our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it. This optimal estimator brings significantly improved performance in both evaluation and learning, and minimizes data requirements. Empirical observations corroborate our theoretical findings.