# Deep anytime-valid hypothesis testing

arXiv: https://arxiv.org/abs/2310.19384 · HTML: https://mlsanity.com/paper/2310.19384/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Safe Testing](https://mlsanity.com/paper/1906.07801/) (2019)
- [Fast Two-Sample Testing with Analytic Representations of Probability Measures](https://mlsanity.com/paper/1506.04725/) (2015)
- [There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"](https://mlsanity.com/paper/2102.05569/) (2021)
- [Classification accuracy as a proxy for two sample testing](https://mlsanity.com/paper/1602.02210/) (2016)
- [Nonparametric Two-Sample Testing by Betting](https://mlsanity.com/paper/2112.09162/) (2021)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [Sequential Kernelized Independence Testing](https://mlsanity.com/paper/2212.07383/) (2022)
- [Model-X Sequential Testing for Conditional Independence via Testing by Betting](https://mlsanity.com/paper/2210.00354/) (2022)
- [E-Valuating Classifier Two-Sample Tests](https://mlsanity.com/paper/2210.13027/) (2022)

## Built on by (most cited first, 7 of 7)

- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [An Auditing Test To Detect Behavioral Shift in Language Models](https://mlsanity.com/paper/2410.19406/) (2024)
- [Optimistic Interior Point Methods for Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2502.07774/) (2025)
- [A complete characterization of sequential testability and change detectability in i.i.d. models](https://mlsanity.com/paper/2609.05752/) (2026)
- [Sequential Kernel-based Conditional Independence Testing via Adaptive Betting](https://mlsanity.com/paper/2606.18993/) (2026)
- [Evidence-Gated Research: Statistically Controlled Model Adoption in Adaptive Search](https://mlsanity.com/paper/2610.01751/) (2026)

## Abstract (arXiv)

We propose a general framework for constructing powerful, sequential hypothesis tests for a large class of nonparametric testing problems. The null hypothesis for these problems is defined in an abstract form using the action of two known operators on the data distribution. This abstraction allows for a unified treatment of several classical tasks, such as two-sample testing, independence testing, and conditional-independence testing, as well as modern problems, such as testing for adversarial robustness of machine learning (ML) models. Our proposed framework has the following advantages over classical batch tests: 1) it continuously monitors online data streams and efficiently aggregates evidence against the null, 2) it provides tight control over the type I error without the need for multiple testing correction, 3) it adapts the sample size requirement to the unknown hardness of the problem. We develop a principled approach of leveraging the representation capability of ML models within the testing-by-betting framework, a game-theoretic approach for designing sequential tests. Empirical results on synthetic and real-world datasets demonstrate that tests instantiated using our general framework are competitive against specialized baselines on several tasks.