# Safe Testing

arXiv: https://arxiv.org/abs/1906.07801 · HTML: https://mlsanity.com/paper/1906.07801/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 30 of 101)

- [Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio](https://mlsanity.com/paper/2110.14099/) (2021)
- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [E-detectors: a nonparametric framework for sequential change detection](https://mlsanity.com/paper/2203.03532/) (2022)
- [A unified recipe for deriving (time-uniform) PAC-Bayes bounds](https://mlsanity.com/paper/2302.03421/) (2023)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [Enhancing Conformal Prediction Using E-Test Statistics](https://mlsanity.com/paper/2403.19082/) (2024)
- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [Sequential Kernelized Independence Testing](https://mlsanity.com/paper/2212.07383/) (2022)
- [Principal-Agent Hypothesis Testing](https://mlsanity.com/paper/2205.06812/) (2022)
- [Model-X Sequential Testing for Conditional Independence via Testing by Betting](https://mlsanity.com/paper/2210.00354/) (2022)
- [Active, anytime-valid risk controlling prediction sets](https://mlsanity.com/paper/2406.10490/) (2024)
- [Backward Conformal Prediction](https://mlsanity.com/paper/2505.13732/) (2025)
- [Online multiple testing with e-values](https://mlsanity.com/paper/2311.06412/) (2023)
- [E-Valuating Classifier Two-Sample Tests](https://mlsanity.com/paper/2210.13027/) (2022)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations](https://mlsanity.com/paper/2212.14411/) (2022)
- [Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams](https://mlsanity.com/paper/2402.06122/) (2024)
- [A unified framework for bandit multiple testing](https://mlsanity.com/paper/2107.07322/) (2021)
- [Comparing Sequential Forecasters](https://mlsanity.com/paper/2110.00115/) (2021)
- [Multi-disciplinary fairness considerations in machine learning for clinical trials](https://mlsanity.com/paper/2205.08875/) (2022)
- [Huber-Robust Confidence Sequences](https://mlsanity.com/paper/2301.09573/) (2023)
- [Deep anytime-valid hypothesis testing](https://mlsanity.com/paper/2310.19384/) (2023)
- [Resolution Diagnostics for Paired LLM Evaluation](https://mlsanity.com/paper/2605.30315/) (2026)
- [Nonparametric extensions of randomized response for private confidence sets](https://mlsanity.com/paper/2202.08728/) (2022)
- [Risk-limiting Financial Audits via Weighted Sampling without Replacement](https://mlsanity.com/paper/2305.06884/) (2023)
- [Post-detection inference for sequential changepoint localization](https://mlsanity.com/paper/2502.06096/) (2025)
- [Adaptive Coverage Policies in Conformal Prediction](https://mlsanity.com/paper/2510.04318/) (2025)
- [Conformal testing in a binary model situation](https://mlsanity.com/paper/2104.01885/) (2021)
- [Optimistic Interior Point Methods for Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2502.07774/) (2025)

## Abstract (arXiv)

We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test and the 2 x 2 contingency table. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools.