# Sequential Predictive Two-Sample and Independence Testing

arXiv: https://arxiv.org/abs/2305.00143 · HTML: https://mlsanity.com/paper/2305.00143/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 29 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [Black-Box Reductions for Parameter-free Online Learning in Banach Spaces](https://mlsanity.com/paper/1802.06293/) (2018)
- [Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy](https://mlsanity.com/paper/1611.04488/) (2016)
- [Interpretable Distribution Features with Maximum Testing Power](https://mlsanity.com/paper/1605.06796/) (2016)
- [There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"](https://mlsanity.com/paper/2102.05569/) (2021)
- [Admissible anytime-valid sequential inference must rely on nonnegative martingales](https://mlsanity.com/paper/2009.03167/) (2020)
- [Universal Inference](https://mlsanity.com/paper/1912.11436/) (2019)
- [Classification accuracy as a proxy for two sample testing](https://mlsanity.com/paper/1602.02210/) (2016)
- [Nonparametric Two-Sample Testing by Betting](https://mlsanity.com/paper/2112.09162/) (2021)
- [Classification Logit Two-sample Testing by Neural Networks](https://mlsanity.com/paper/1909.11298/) (2019)
- [Nonparametric independence testing via mutual information](https://mlsanity.com/paper/1711.06642/) (2017)
- [AutoML Two-Sample Test](https://mlsanity.com/paper/2206.08843/) (2022)
- [Sequential Kernelized Independence Testing](https://mlsanity.com/paper/2212.07383/) (2022)
- [Dimension-agnostic inference using cross U-statistics](https://mlsanity.com/paper/2011.05068/) (2020)
- [Model-X Sequential Testing for Conditional Independence via Testing by Betting](https://mlsanity.com/paper/2210.00354/) (2022)
- [On the Use of Random Forest for Two-Sample Testing](https://mlsanity.com/paper/1903.06287/) (2019)
- [Global and Local Two-Sample Tests via Regression](https://mlsanity.com/paper/1812.08927/) (2018)
- [Anytime Valid Tests of Conditional Independence Under Model-X](https://mlsanity.com/paper/2209.12637/) (2022)
- [E-Valuating Classifier Two-Sample Tests](https://mlsanity.com/paper/2210.13027/) (2022)
- [Low-Complexity Nonparametric Bayesian Online Prediction with Universal Guarantees](https://mlsanity.com/paper/1901.07662/) (2019)

## Built on by (most cited first, 21 of 21)

- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [Prediction-Powered E-Values](https://mlsanity.com/paper/2502.04294/) (2025)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Active, anytime-valid risk controlling prediction sets](https://mlsanity.com/paper/2406.10490/) (2024)
- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [An Auditing Test To Detect Behavioral Shift in Language Models](https://mlsanity.com/paper/2410.19406/) (2024)
- [Deep anytime-valid hypothesis testing](https://mlsanity.com/paper/2310.19384/) (2023)
- [Optimistic Interior Point Methods for Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2502.07774/) (2025)
- [Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://mlsanity.com/paper/2602.13848/) (2026)
- [Combining Evidence Across Filtrations](https://mlsanity.com/paper/2402.09698/) (2024)
- [Optimal Anytime-Valid Tests for Composite Nulls](https://mlsanity.com/paper/2512.20039/) (2025)
- [Multi-Armed Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2603.17925/) (2026)
- [A complete characterization of sequential testability and change detectability in i.i.d. models](https://mlsanity.com/paper/2609.05752/) (2026)
- [Kernel conditional tests from learning-theoretic bounds](https://mlsanity.com/paper/2506.03898/) (2025)
- [Matched-Pair Experimental Design with Active Learning](https://mlsanity.com/paper/2509.10742/) (2025)
- [Learning to Test: Latent Representation Learning for Dynamic Instability Detection](https://mlsanity.com/paper/2604.10967/) (2026)
- [Optimal sequential tests yield log-optimal e-processes](https://mlsanity.com/paper/2605.12720/) (2026)
- [The Price of Peeking: Anytime-Valid Leakage Detection on ML-KEM EM Traces](https://mlsanity.com/paper/2609.33597/) (2026)
- [Always-On Experimentation](https://mlsanity.com/paper/2609.38695/) (2026)
- [SLICE: Sequential Likelihood-Free Inference via Conformal E-Values](https://mlsanity.com/paper/2610.13855/) (2026)

## Abstract (arXiv)

We study the problems of sequential nonparametric two-sample and independence testing. Sequential tests process data online and allow using observed data to decide whether to stop and reject the null hypothesis or to collect more data, while maintaining type I error control. We build upon the principle of (nonparametric) testing by betting, where a gambler places bets on future observations and their wealth measures evidence against the null hypothesis. While recently developed kernel-based betting strategies often work well on simple distributions, selecting a suitable kernel for high-dimensional or structured data, such as images, is often nontrivial. To address this drawback, we design prediction-based betting strategies that rely on the following fact: if a sequentially updated predictor starts to consistently determine (a) which distribution an instance is drawn from, or (b) whether an instance is drawn from the joint distribution or the product of the marginal distributions (the latter produced by external randomization), it provides evidence against the two-sample or independence nulls respectively. We empirically demonstrate the superiority of our tests over kernel-based approaches under structured settings. Our tests can be applied beyond the case of independent and identically distributed data, remaining valid and powerful even when the data distribution drifts over time.