# Off-policy Confidence Sequences

arXiv: https://arxiv.org/abs/2102.09540 · HTML: https://mlsanity.com/paper/2102.09540/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Time-uniform, nonparametric, nonasymptotic confidence sequences](https://mlsanity.com/paper/1810.08240/) (2018)
- [Coin Betting and Parameter-Free Online Learning](https://mlsanity.com/paper/1602.04128/) (2016)
- [Black-Box Reductions for Parameter-free Online Learning in Banach Spaces](https://mlsanity.com/paper/1802.06293/) (2018)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Follow the Leader If You Can, Hedge If You Must](https://mlsanity.com/paper/1301.0534/) (2013)
- [CoinDICE: Off-Policy Confidence Interval Estimation](https://mlsanity.com/paper/2010.11652/) (2020)
- [Admissible anytime-valid sequential inference must rely on nonnegative martingales](https://mlsanity.com/paper/2009.03167/) (2020)
- [Parameter-Free Online Convex Optimization with Sub-Exponential Noise](https://mlsanity.com/paper/1902.01500/) (2019)
- [Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1906.03735/) (2019)
- [On Equivalence of Martingale Tail Bounds and Deterministic Regret Inequalities](https://mlsanity.com/paper/1510.03925/) (2015)
- [Empirical Likelihood for Contextual Bandits](https://mlsanity.com/paper/1906.03323/) (2019)

## Built on by (most cited first, 15 of 15)

- [Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio](https://mlsanity.com/paper/2110.14099/) (2021)
- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [Martingale Methods for Sequential Estimation of Convex Functionals and Divergences](https://mlsanity.com/paper/2103.09267/) (2021)
- [Comparing Sequential Forecasters](https://mlsanity.com/paper/2110.00115/) (2021)
- [Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effect](https://mlsanity.com/paper/2411.14341/) (2024)
- [Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezing](https://mlsanity.com/paper/2502.10826/) (2025)
- [Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability](https://mlsanity.com/paper/2607.08349/) (2026)
- [Offline Policy Comparison under Limited Historical Agent-Environment Interactions](https://mlsanity.com/paper/2106.03934/) (2021)
- [Inference on Optimal Dynamic Policies via Softmax Approximation](https://mlsanity.com/paper/2303.04416/) (2023)
- [Off-policy estimation with adaptively collected data: the power of online learning](https://mlsanity.com/paper/2411.12786/) (2024)
- [A lower confidence sequence for the changing mean of non-negative right heavy-tailed observations with bounded mean](https://mlsanity.com/paper/2210.11133/) (2022)
- [Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Bounds](https://mlsanity.com/paper/2609.17560/) (2026)
- [Anytime-valid Optimal Policy Identification](https://mlsanity.com/paper/2606.17515/) (2026)
- [Policy Learning with Abstention](https://mlsanity.com/paper/2510.19672/) (2025)
- [What Do We Care About in Bandits with Noncompliance? BRACE: Bandits with Recommendations, Abstention, and Certified Effects](https://mlsanity.com/paper/2603.09532/) (2026)

## Abstract (arXiv)

We develop confidence bounds that hold uniformly over time for off-policy evaluation in the contextual bandit setting. These confidence sequences are based on recent ideas from martingale analysis and are non-asymptotic, non-parametric, and valid at arbitrary stopping times. We provide algorithms for computing these confidence sequences that strike a good balance between computational and statistical efficiency. We empirically demonstrate the tightness of our approach in terms of failure probability and width and apply it to the "gated deployment" problem of safely upgrading a production contextual bandit system.