# Testing For Distribution Shifts with Conditional Conformal Test Martingales

arXiv: https://arxiv.org/abs/2602.13848 · HTML: https://mlsanity.com/paper/2602.13848/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 14 Feb 2026.

## Builds on (most cited first, 30 of 33)

- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Black-Box Reductions for Parameter-free Online Learning in Banach Spaces](https://mlsanity.com/paper/1802.06293/) (2018)
- [There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"](https://mlsanity.com/paper/2102.05569/) (2021)
- [Universal Inference](https://mlsanity.com/paper/1912.11436/) (2019)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Nonparametric Two-Sample Testing by Betting](https://mlsanity.com/paper/2112.09162/) (2021)
- [Testing randomness](https://mlsanity.com/paper/1906.09256/) (2019)
- [Testing exchangeability: fork-convexity, supermartingales, and e-processes](https://mlsanity.com/paper/2102.00630/) (2021)
- [Retrain or not retrain: Conformal test martingales for change-point detection](https://mlsanity.com/paper/2102.10439/) (2021)
- [E-detectors: a nonparametric framework for sequential change detection](https://mlsanity.com/paper/2203.03532/) (2022)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [Inductive Conformal Martingales for Change-Point Detection](https://mlsanity.com/paper/1706.03415/) (2017)
- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Testing for concept shift online](https://mlsanity.com/paper/2012.14246/) (2020)
- [Model-X Sequential Testing for Conditional Independence via Testing by Betting](https://mlsanity.com/paper/2210.00354/) (2022)
- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [Monitoring Risks in Test-Time Adaptation](https://mlsanity.com/paper/2507.08721/) (2025)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [Synthetic-Powered Predictive Inference](https://mlsanity.com/paper/2505.13432/) (2025)
- [On the optimality of coin-betting for mean estimation](https://mlsanity.com/paper/2412.02640/) (2024)

## Built on by (most cited first, 4 of 4)

- [Distribution-free changepoint localization after sequential change detection](https://mlsanity.com/paper/2606.01256/) (2026)
- [Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG](https://mlsanity.com/paper/2605.29139/) (2026)
- [The Price of Self-Calibration: Exact Evidence Budgets and Manufactured Blind Sets in Adaptive Monitoring](https://mlsanity.com/paper/2609.23254/) (2026)
- [Monitoring for Shifts at Unknown Times: Parameter-Free Conditional Conformal Martingales](https://mlsanity.com/paper/2610.17343/) (2026)

## Abstract (arXiv)

We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.