# Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezing

arXiv: https://arxiv.org/abs/2502.10826 · HTML: https://mlsanity.com/paper/2502.10826/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Competing Bandits in Matching Markets](https://mlsanity.com/paper/1906.05363/) (2019)
- [Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio](https://mlsanity.com/paper/2110.14099/) (2021)
- [Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting](https://mlsanity.com/paper/2006.10460/) (2020)
- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequality](https://mlsanity.com/paper/2212.09900/) (2022)
- [Bayesian Counterfactual Risk Minimization](https://mlsanity.com/paper/1806.11500/) (2018)
- [Confidence sequences for sampling without replacement](https://mlsanity.com/paper/2006.04347/) (2020)
- [PAC-Bayesian Offline Contextual Bandits With Guarantees](https://mlsanity.com/paper/2210.13132/) (2022)
- [Catoni-style confidence sequences for heavy-tailed mean estimation](https://mlsanity.com/paper/2202.01250/) (2022)
- [Efron-Stein PAC-Bayesian Inequalities](https://mlsanity.com/paper/1909.01931/) (2019)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Off-policy Confidence Sequences](https://mlsanity.com/paper/2102.09540/) (2021)
- [Exponential Smoothing for Off-Policy Learning](https://mlsanity.com/paper/2305.15877/) (2023)
- [Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits](https://mlsanity.com/paper/2306.07923/) (2023)
- [Importance-Weighted Offline Learning Done Right](https://mlsanity.com/paper/2309.15771/) (2023)
- [On Confidence Sequences for Bounded Random Processes via Universal Gambling Strategies](https://mlsanity.com/paper/2207.12382/) (2022)
- [Empirical Likelihood for Contextual Bandits](https://mlsanity.com/paper/1906.03323/) (2019)
- [Sequential Counterfactual Risk Minimization](https://mlsanity.com/paper/2302.12120/) (2023)
- [Gambling-Based Confidence Sequences for Bounded Random Vectors](https://mlsanity.com/paper/2402.03683/) (2024)

## Built on by (most cited first, 4 of 4)

- [Multi-Armed Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2603.17925/) (2026)
- [Exploration-Exploitation Tradeoff in Universal Lossy Compression](https://mlsanity.com/paper/2506.20261/) (2025)
- [Always-On Experimentation](https://mlsanity.com/paper/2609.38695/) (2026)
- [Efficient Best-of-N policy evaluation for inference-time alignment](https://mlsanity.com/paper/2610.09250/) (2026)

## Abstract (arXiv)

We consider off-policy selection and learning in contextual bandits, where the learner aims to select or train a reward-maximizing policy using data collected by a fixed behavior policy. Our contribution is two-fold. First, we propose a novel off-policy selection method that leverages a new betting-based confidence bound applied to an inverse propensity weight sequence. Our theoretical analysis reveals that this method achieves a significantly improved, variance-adaptive guarantee over prior work. Second, we propose a novel and generic condition on the optimization objective for off-policy learning that strikes a different balance between bias and variance. One special case, which we call freezing, tends to induce low variance, which is preferred in small-data regimes. Our analysis shows that it matches the best existing guarantees. In our empirical study, our selection method outperforms existing methods, and freezing exhibits improved performance in small-sample regimes.