# Post-Contextual-Bandit Inference

arXiv: https://arxiv.org/abs/2106.00418 · HTML: https://mlsanity.com/paper/2106.00418/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Confidence Intervals for Policy Evaluation in Adaptive Experiments](https://mlsanity.com/paper/1911.02768/) (2019)
- [Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning](https://mlsanity.com/paper/1909.05850/) (2019)
- [Balanced Policy Evaluation and Learning](https://mlsanity.com/paper/1705.07384/) (2017)
- [Statistical Inference with M-Estimators on Adaptively Collected Data](https://mlsanity.com/paper/2104.14074/) (2021)
- [Why Adaptively Collected Data Have Negative Bias and How to Correct for It](https://mlsanity.com/paper/1708.01977/) (2017)
- [Dynamic Pricing with Demand Covariates](https://mlsanity.com/paper/1604.07463/) (2016)
- [Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1906.03735/) (2019)
- [On the bias, risk and consistency of sample means in multi-armed bandits](https://mlsanity.com/paper/1902.00746/) (2019)
- [Estimation Considerations in Contextual Bandits](https://mlsanity.com/paper/1711.07077/) (2017)
- [Dynamic Assortment Personalization in High Dimensions](https://mlsanity.com/paper/1610.05604/) (2016)

## Built on by (most cited first, 30 of 32)

- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [UCB algorithms for multi-armed bandits: Precise regret and adaptive inference](https://mlsanity.com/paper/2412.06126/) (2024)
- [Counterfactual inference in sequential experiments](https://mlsanity.com/paper/2202.06891/) (2022)
- [Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices](https://mlsanity.com/paper/2403.03589/) (2024)
- [Inference with the Upper Confidence Bound Algorithm](https://mlsanity.com/paper/2408.04595/) (2024)
- [Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations](https://mlsanity.com/paper/2212.14411/) (2022)
- [Synthetically Controlled Bandits](https://mlsanity.com/paper/2202.07079/) (2022)
- [Stable Thompson Sampling: Valid Inference via Variance Inflation](https://mlsanity.com/paper/2505.23260/) (2025)
- [Semi-parametric inference based on adaptively collected data](https://mlsanity.com/paper/2303.02534/) (2023)
- [Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resampling](https://mlsanity.com/paper/2304.05365/) (2023)
- [Adaptive Linear Estimating Equations](https://mlsanity.com/paper/2307.07320/) (2023)
- [Combining Experimental and Historical Data for Policy Evaluation](https://mlsanity.com/paper/2406.00317/) (2024)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning](https://mlsanity.com/paper/2110.15501/) (2021)
- [Optimal Best Arm Identification in Two-Armed Bandits with a Fixed Budget under a Small Gap](https://mlsanity.com/paper/2201.04469/) (2022)
- [Online learning in bandits with predicted context](https://mlsanity.com/paper/2307.13916/) (2023)
- [Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification](https://mlsanity.com/paper/2509.14218/) (2025)
- [Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization](https://mlsanity.com/paper/2605.10405/) (2026)
- [Best Arm Identification with Contextual Information under a Small Gap](https://mlsanity.com/paper/2209.07330/) (2022)
- [Simulation-Based Inference for Adaptive Experiments](https://mlsanity.com/paper/2506.02881/) (2025)
- [Statistical Inference in Reinforcement Learning: A Selective Survey](https://mlsanity.com/paper/2502.16195/) (2025)
- [Kernel Treatment Effects with Adaptively Collected Data](https://mlsanity.com/paper/2510.10245/) (2025)
- [A Statistical Test for the Benefits of Personalizing Interventions](https://mlsanity.com/paper/2607.08951/) (2026)
- [Entropy Regularization for Population Estimation](https://mlsanity.com/paper/2208.11747/) (2022)
- [Demistifying Inference after Adaptive Experiments](https://mlsanity.com/paper/2405.01281/) (2024)
- [Efficient Inference after Directionally Stable Adaptive Experiments](https://mlsanity.com/paper/2602.21478/) (2026)
- [Anytime-valid Optimal Policy Identification](https://mlsanity.com/paper/2606.17515/) (2026)
- [SNPL: Simultaneous Policy Learning and Evaluation for Safe Multi-Objective Policy Improvement](https://mlsanity.com/paper/2503.12760/) (2025)
- [Bernstein-von Mises for Adaptively Collected Data](https://mlsanity.com/paper/2511.06639/) (2025)
- [Fixed-Horizon Self-Normalized Inference for Adaptive Experiments via Martingale AIPW/DML with Logged Propensities](https://mlsanity.com/paper/2602.15559/) (2026)

## Abstract (arXiv)

Contextual bandit algorithms are increasingly replacing non-adaptive A/B tests in e-commerce, healthcare, and policymaking because they can both improve outcomes for study participants and increase the chance of identifying good or even best policies. To support credible inference on novel interventions at the end of the study, nonetheless, we still want to construct valid confidence intervals on average treatment effects, subgroup effects, or value of new policies. The adaptive nature of the data collected by contextual bandit algorithms, however, makes this difficult: standard estimators are no longer asymptotically normally distributed and classic confidence intervals fail to provide correct coverage. While this has been addressed in non-contextual settings by using stabilized estimators, the contextual setting poses unique challenges that we tackle for the first time in this paper. We propose the Contextual Adaptive Doubly Robust (CADR) estimator, the first estimator for policy value that is asymptotically normal under contextual adaptive data collection. The main technical challenge in constructing CADR is designing adaptive and consistent conditional standard deviation estimators for stabilization. Extensive numerical experiments using 57 OpenML datasets demonstrate that confidence intervals based on CADR uniquely provide correct coverage.