# Retrain or not retrain: Conformal test martingales for change-point detection

arXiv: https://arxiv.org/abs/2102.10439 · HTML: https://mlsanity.com/paper/2102.10439/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 15 references that are in the map.
- First posted on arXiv 20 Feb 2021.

## Builds on (most cited first, 1 of 1)

- [Testing randomness](https://mlsanity.com/paper/1906.09256/) (2019)

## Built on by (most cited first, 30 of 32)

- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [Evaluating Machine Translation Quality with Conformal Predictive Distributions](https://mlsanity.com/paper/2306.01549/) (2023)
- [A unified framework for bandit multiple testing](https://mlsanity.com/paper/2107.07322/) (2021)
- [Online Distribution Shift Detection via Recency Prediction](https://mlsanity.com/paper/2211.09916/) (2022)
- [Test for non-negligible adverse shifts](https://mlsanity.com/paper/2107.02990/) (2021)
- [Conformal Prediction for Time-series Forecasting with Change Points](https://mlsanity.com/paper/2509.02844/) (2025)
- [Conformal testing in a binary model situation](https://mlsanity.com/paper/2104.01885/) (2021)
- [Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors](https://mlsanity.com/paper/2402.16388/) (2024)
- [Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://mlsanity.com/paper/2602.13848/) (2026)
- [Combining Evidence Across Filtrations](https://mlsanity.com/paper/2402.09698/) (2024)
- [Diagnostic Runtime Monitoring with Martingales](https://mlsanity.com/paper/2407.21748/) (2024)
- [Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts](https://mlsanity.com/paper/2602.02229/) (2026)
- [Distribution-free changepoint localization after sequential change detection](https://mlsanity.com/paper/2606.01256/) (2026)
- [Temporal Generalization Estimation in Evolving Graphs](https://mlsanity.com/paper/2404.04969/) (2024)
- [Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors](https://mlsanity.com/paper/2409.15548/) (2024)
- [Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG](https://mlsanity.com/paper/2605.29139/) (2026)
- [Enhancement of prediction algorithms by betting](https://mlsanity.com/paper/2105.08669/) (2021)
- [Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with nonconform](https://mlsanity.com/paper/2605.13642/) (2026)
- [Betting on Moments: Legendre Jumper Martingales for Online Exchangeability Testing](https://mlsanity.com/paper/2606.20859/) (2026)
- [Towards Practicable Sequential Shift Detectors](https://mlsanity.com/paper/2307.14758/) (2023)
- [Conformal Blindness: A Note on A-Cryptic change-points](https://mlsanity.com/paper/2601.01147/) (2026)
- [When the Martingale Never Stops Firing: Anytime-Valid Gating on Real Forecast Streams](https://mlsanity.com/paper/2608.30502/) (2026)
- [Strategic Self-Consistency](https://mlsanity.com/paper/2609.30352/) (2026)
- [The Premise Is the Problem: Exchangeability Failure in Self-Monitored Test-Time Adaptation](https://mlsanity.com/paper/2610.07038/) (2026)
- [The Limits of Query Monitoring against Distributed Model Extraction](https://mlsanity.com/paper/2610.13083/) (2026)

## Abstract (arXiv)

We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.