mlsanityData to 7 Oct 2026

Off-policy Confidence Sequences

arXiv 2102.09540 ↗First posted 18 Feb 2021cs.LG math.ST stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

We develop confidence bounds that hold uniformly over time for off-policy evaluation in the contextual bandit setting. These confidence sequences are based on recent ideas from martingale analysis and are non-asymptotic, non-parametric, and valid at arbitrary stopping times. We provide algorithms for computing these confidence sequences that strike a good balance between computational and statistical efficiency. We empirically demonstrate the tightness of our approach in terms of failure probability and width and apply it to the "gated deployment" problem of safely upgrading a production contextual bandit system.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 12 of its 36 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

12 of 36 references.

References · 12

  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  2. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  3. 2016Coin Betting and Parameter-Free Online Learningcited by 131
  4. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 124
  5. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  6. 2013Follow the Leader If You Can, Hedge If You Mustcited by 68
  7. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  8. 2020Admissible anytime-valid sequential inference must rely on nonnegative martingalescited by 43
  9. 2019Parameter-Free Online Convex Optimization with Sub-Exponential Noisecited by 28
  10. 2019Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learningcited by 28
  11. 2015On Equivalence of Martingale Tail Bounds and Deterministic Regret Inequalitiescited by 27
  12. 2019Empirical Likelihood for Contextual Banditscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 1; 2024: 2; 2025: 2; 2026: 4.

15 papers.

Sort:

  1. 2021Tight Concentrations and Confidence Sequences from the Regret of Universal Portfoliocited by 42
  2. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  3. 2021Martingale Methods for Sequential Estimation of Convex Functionals and Divergencescited by 17
  4. 2021Comparing Sequential Forecasterscited by 8
  5. 2021Offline Policy Comparison under Limited Historical Agent-Environment Interactionscited by 3
  6. 2023Inference on Optimal Dynamic Policies via Softmax Approximationcited by 3
  7. 2024Off-policy estimation with adaptively collected data: the power of online learningcited by 3
  8. 2024Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effectcited by 3
  9. 2025Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezingcited by 3
  10. 2022A lower confidence sequence for the changing mean of non-negative right heavy-tailed observations with bounded meancited by 2
  11. 2026Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretabilitycited by 2
  12. 2026Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Boundscited by 2
3 more
  1. 2026Anytime-valid Optimal Policy Identificationcited by 1
  2. 2025Policy Learning with Abstentioncited by 0
  3. 2026What Do We Care About in Bandits with Noncompliance? BRACE: Bandits with Recommendations, Abstention, and Certified Effectscited by 0