# Always-On Experimentation

arXiv: https://arxiv.org/abs/2609.38695 · HTML: https://mlsanity.com/paper/2609.38695/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 68 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 30 of 34)

- [Time-uniform, nonparametric, nonasymptotic confidence sequences](https://mlsanity.com/paper/1810.08240/) (2018)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [Safe Testing](https://mlsanity.com/paper/1906.07801/) (2019)
- [Hypothesis testing with e-values](https://mlsanity.com/paper/2410.23614/) (2024)
- [Admissible anytime-valid sequential inference must rely on nonnegative martingales](https://mlsanity.com/paper/2009.03167/) (2020)
- [Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio](https://mlsanity.com/paper/2110.14099/) (2021)
- [Nonparametric Two-Sample Testing by Betting](https://mlsanity.com/paper/2112.09162/) (2021)
- [Assessing Time-Varying Causal Effect Moderation in Mobile Health](https://mlsanity.com/paper/1601.00237/) (2016)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Online Rules for Control of False Discovery Rate and False Discovery Exceedance](https://mlsanity.com/paper/1603.09000/) (2016)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [A framework for Multi-A(rmed)/B(andit) testing with online FDR control](https://mlsanity.com/paper/1706.05378/) (2017)
- [Sequential Kernelized Independence Testing](https://mlsanity.com/paper/2212.07383/) (2022)
- [Online multiple testing with e-values](https://mlsanity.com/paper/2311.06412/) (2023)
- [The numeraire e-variable and reverse information projection](https://mlsanity.com/paper/2402.18810/) (2024)
- [SAFFRON: an adaptive algorithm for online control of the false discovery rate](https://mlsanity.com/paper/1802.09098/) (2018)
- [ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls](https://mlsanity.com/paper/1905.11465/) (2019)
- [Asynchronous Online Testing of Multiple Hypotheses](https://mlsanity.com/paper/1812.05068/) (2018)
- [Panel Experiments and Dynamic Causal Effects: A Finite Population Perspective](https://mlsanity.com/paper/2003.09915/) (2020)
- [On Confidence Sequences for Bounded Random Processes via Universal Gambling Strategies](https://mlsanity.com/paper/2207.12382/) (2022)
- [On the optimality of coin-betting for mean estimation](https://mlsanity.com/paper/2412.02640/) (2024)
- [On Online Control of False Discovery Rate](https://mlsanity.com/paper/1502.06197/) (2015)
- [E-backtesting](https://mlsanity.com/paper/2209.00991/) (2022)
- [Test Martingales for bounded random variables](https://mlsanity.com/paper/1801.09418/) (2018)
- [An online generalization of the (e-)Benjamini-Hochberg procedure](https://mlsanity.com/paper/2407.20683/) (2024)
- [Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezing](https://mlsanity.com/paper/2502.10826/) (2025)
- [Demonstration Experiments](https://mlsanity.com/paper/2603.06941/) (2026)
- [Sequential Tests of Multiple Hypotheses Controlling False Discovery and Nondiscovery Rates](https://mlsanity.com/paper/1311.3350/) (2013)
- [Online control of the False Discovery Rate in group-sequential platform trials](https://mlsanity.com/paper/2112.10619/) (2021)

## Abstract (arXiv)

Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.