# Always Valid Inference: Bringing Sequential Analysis to A/B Testing

arXiv: https://arxiv.org/abs/1512.04922 · HTML: https://mlsanity.com/paper/1512.04922/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2015.

## Built on by (most cited first, 20 of 20)

- [A/B/n Testing with Control in the Presence of Subpopulations](https://mlsanity.com/paper/2110.15573/) (2021)
- [Sequential Harmful Shift Detection Without Labels](https://mlsanity.com/paper/2412.12910/) (2024)
- [Comparing Sequential Forecasters](https://mlsanity.com/paper/2110.00115/) (2021)
- [Huber-Robust Confidence Sequences](https://mlsanity.com/paper/2301.09573/) (2023)
- [Risk-limiting Financial Audits via Weighted Sampling without Replacement](https://mlsanity.com/paper/2305.06884/) (2023)
- [Anytime-Valid Inference for Double/Debiased Machine Learning of Causal Parameters](https://mlsanity.com/paper/2408.09598/) (2024)
- [The Lifecycle of a Statistical Model: Model Failure Detection, Identification, and Refitting](https://mlsanity.com/paper/2202.04166/) (2022)
- [Searching for a higher power in the human evaluation of MT](https://mlsanity.com/paper/2210.11612/) (2022)
- [Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring](https://mlsanity.com/paper/2304.00420/) (2023)
- [Adaptive Experimentation with Delayed Binary Feedback](https://mlsanity.com/paper/2202.00846/) (2022)
- [Catoni-style Confidence Sequences under Infinite Variance](https://mlsanity.com/paper/2208.03185/) (2022)
- [Adaptive Identification of Populations with Treatment Benefit in Clinical Trials: Machine Learning Challenges and Solutions](https://mlsanity.com/paper/2208.05844/) (2022)
- [A Two-armed Bandit Framework for A/B Testing](https://mlsanity.com/paper/2507.18118/) (2025)
- [Privacy Preserving Adaptive Experiment Design](https://mlsanity.com/paper/2401.08224/) (2024)
- [AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems](https://mlsanity.com/paper/2609.04871/) (2026)
- [On the Tightness and Computational Tractability of Higher-Dimensional Confidence Sequences](https://mlsanity.com/paper/2610.03727/) (2026)
- [Always Valid Risk Monitoring for Online Matrix Completion](https://mlsanity.com/paper/2211.10363/) (2022)
- [Rapid and Scalable Bayesian AB Testing](https://mlsanity.com/paper/2307.14628/) (2023)
- [Beyond ATE: Multi-Criteria Design for A/B Testing](https://mlsanity.com/paper/2509.05864/) (2025)
- [GAAVI: Global Asymptotic Anytime Valid Inference for the Conditional Mean Function](https://mlsanity.com/paper/2602.08096/) (2026)

## Abstract (arXiv)

A/B tests are typically analyzed via frequentist p-values and confidence intervals; but these inferences are wholly unreliable if users endogenously choose samples sizes by *continuously monitoring* their tests. We define *always valid* p-values and confidence intervals that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid inference can be interpreted as a natural interface for a sequential hypothesis test, which empowers users to implement a modified test tailored to them. In particular, we show in an appropriate sense that the measures we develop tradeoff sample size and power efficiently, despite a lack of prior knowledge of the user's relative preference between these two goals. We also use always valid p-values to obtain multiple hypothesis testing control in the sequential context. Our methodology has been implemented in a large scale commercial A/B testing platform to analyze hundreds of thousands of experiments to date.