Always-On Experimentation
Not on the map. Why?
Abstract (arXiv)
Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
34 of 68 references.
References · 34
- 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 229
- 2020Estimating means of bounded random variables by bettingcited by 169
- 2022Game-theoretic statistics and safe anytime-valid inferencecited by 155
- 2019Safe Testingcited by 101
- 2024Hypothesis testing with e-valuescited by 74
- 2020Admissible anytime-valid sequential inference must rely on nonnegative martingalescited by 48
- 2021Tight Concentrations and Confidence Sequences from the Regret of Universal Portfoliocited by 43
- 2021Nonparametric Two-Sample Testing by Bettingcited by 37
- 2016Assessing Time-Varying Causal Effect Moderation in Mobile Healthcited by 27
- 2023Auditing Fairness by Bettingcited by 25
- 2016Online Rules for Control of False Discovery Rate and False Discovery Exceedancecited by 21
- 2023Sequential Predictive Two-Sample and Independence Testingcited by 21
22 more
- 2017A framework for Multi-A(rmed)/B(andit) testing with online FDR controlcited by 17
- 2022Sequential Kernelized Independence Testingcited by 16
- 2023Online multiple testing with e-valuescited by 12
- 2024The numeraire e-variable and reverse information projectioncited by 12
- 2018SAFFRON: an adaptive algorithm for online control of the false discovery ratecited by 11
- 2019ADDIS: an adaptive discarding algorithm for online FDR control with conservative nullscited by 11
- 2018Asynchronous Online Testing of Multiple Hypothesescited by 10
- 2020Panel Experiments and Dynamic Causal Effects: A Finite Population Perspectivecited by 8
- 2022On Confidence Sequences for Bounded Random Processes via Universal Gambling Strategiescited by 8
- 2024On the optimality of coin-betting for mean estimationcited by 7
- 2015On Online Control of False Discovery Ratecited by 6
- 2022E-backtestingcited by 6
- 2018Test Martingales for bounded random variablescited by 5
- 2024An online generalization of the (e-)Benjamini-Hochberg procedurecited by 4
- 2025Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezingcited by 4
- 2026Demonstration Experimentscited by 3
- 2013Sequential Tests of Multiple Hypotheses Controlling False Discovery and Nondiscovery Ratescited by 2
- 2021Online control of the False Discovery Rate in group-sequential platform trialscited by 2
- 2024Identification and estimation of causal effects using non-concurrent controls in platform trialscited by 1
- 2026Global Sequential Testing for Multi-Stream Auditingcited by 1
- 2026Multi-Armed Sequential Hypothesis Testing by Bettingcited by 1
- 2026Sample-Efficient Multiple Testing with Adaptive Data Collectioncited by 1
Built on by
Not cited yet.