# Nonparametric Two-Sample Testing by Betting

arXiv: https://arxiv.org/abs/2112.09162 · HTML: https://mlsanity.com/paper/2112.09162/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2021.

## Built on by (most cited first, 30 of 37)

- [E-detectors: a nonparametric framework for sequential change detection](https://mlsanity.com/paper/2203.03532/) (2022)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [Sequential Kernelized Independence Testing](https://mlsanity.com/paper/2212.07383/) (2022)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection](https://mlsanity.com/paper/2409.15844/) (2024)
- [Active, anytime-valid risk controlling prediction sets](https://mlsanity.com/paper/2406.10490/) (2024)
- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [E-Valuating Classifier Two-Sample Tests](https://mlsanity.com/paper/2210.13027/) (2022)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [An Auditing Test To Detect Behavioral Shift in Language Models](https://mlsanity.com/paper/2410.19406/) (2024)
- [Deep anytime-valid hypothesis testing](https://mlsanity.com/paper/2310.19384/) (2023)
- [On Stopping Times of Power-one Sequential Tests: Tight Lower and Upper Bounds](https://mlsanity.com/paper/2504.19952/) (2025)
- [Sequentially Auditing Differential Privacy](https://mlsanity.com/paper/2509.07055/) (2025)
- [Optimistic Interior Point Methods for Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2502.07774/) (2025)
- [Auditing Pay-Per-Token in Large Language Models](https://mlsanity.com/paper/2510.05181/) (2025)
- [Stopping Rules for Stochastic Gradient Descent via Anytime-Valid Confidence Sequences](https://mlsanity.com/paper/2512.13123/) (2025)
- [Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://mlsanity.com/paper/2602.13848/) (2026)
- [Combining Evidence Across Filtrations](https://mlsanity.com/paper/2402.09698/) (2024)
- [I Bet You Did Not Mean That: Testing Semantic Importance via Betting](https://mlsanity.com/paper/2405.19146/) (2024)
- [Sequential Kernelized Stein Discrepancy](https://mlsanity.com/paper/2409.17505/) (2024)
- [Optimal Anytime-Valid Tests for Composite Nulls](https://mlsanity.com/paper/2512.20039/) (2025)
- [Global Sequential Testing for Multi-Stream Auditing](https://mlsanity.com/paper/2602.21479/) (2026)
- [Multi-Armed Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2603.17925/) (2026)
- [Semi-Supervised Hypothesis Testing by Betting on Predictions](https://mlsanity.com/paper/2605.28533/) (2026)
- [A complete characterization of sequential testability and change detectability in i.i.d. models](https://mlsanity.com/paper/2609.05752/) (2026)
- [Kernel conditional tests from learning-theoretic bounds](https://mlsanity.com/paper/2506.03898/) (2025)
- [Matched-Pair Experimental Design with Active Learning](https://mlsanity.com/paper/2509.10742/) (2025)
- [Technical note on Sequential Test-Time Adaptation via Martingale-Driven Fisher Prompting](https://mlsanity.com/paper/2510.03839/) (2025)

## Abstract (arXiv)

We study the problem of designing consistent sequential two-sample tests in a nonparametric setting. Guided by the principle of testing by betting, we reframe this task into that of selecting a sequence of payoff functions that maximize the wealth of a fictitious bettor, betting against the null in a repeated game. In this setting, the relative increase in the bettor's wealth has a precise interpretation as the measure of evidence against the null, and thus our sequential test rejects the null when the wealth crosses an appropriate threshold. We develop a general framework for setting up the betting game for two-sample testing, in which the payoffs are selected by a prediction strategy as data-driven predictable estimates of the witness function associated with the variational representation of some statistical distance measures, such as integral probability metrics (IPMs). We then formally relate the statistical properties of the test~(such as consistency, type-II error exponent and expected sample size) to the regret of the corresponding prediction strategy. We construct a practical sequential two-sample test by instantiating our general strategy with the kernel-MMD metric, and demonstrate its ability to adapt to the difficulty of the unknown alternative through theoretical and empirical results. Our framework is versatile, and easily extends to other problems; we illustrate this by applying our approach to construct consistent tests for the following problems: (i) time-varying two-sample testing with non-exchangeable observations, and (ii) an abstract class of "invariant" testing problems, including symmetry and independence testing.