# Testing for concept shift online

arXiv: https://arxiv.org/abs/2012.14246 · HTML: https://mlsanity.com/paper/2012.14246/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2020.

## Built on by (most cited first, 14 of 14)

- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Fair When Trained, Unfair When Deployed: Observable Fairness Measures are Unstable in Performative Prediction Settings](https://mlsanity.com/paper/2202.05049/) (2022)
- [A unified framework for dataset shift diagnostics](https://mlsanity.com/paper/2205.08340/) (2022)
- [Conformal testing in a binary model situation](https://mlsanity.com/paper/2104.01885/) (2021)
- [Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://mlsanity.com/paper/2602.13848/) (2026)
- [Diagnostic Runtime Monitoring with Martingales](https://mlsanity.com/paper/2407.21748/) (2024)
- [Enhancement of prediction algorithms by betting](https://mlsanity.com/paper/2105.08669/) (2021)
- [Efficient Learning Under Density Shift in Incremental Settings Using Cram'er-Rao-Based Regularization](https://mlsanity.com/paper/2502.12949/) (2025)
- [A Conformal Predictive Measure for Assessing Catastrophic Forgetting](https://mlsanity.com/paper/2505.10677/) (2025)
- [Adapting to Fragmented and Evolving Data: A Fisher Information Perspective](https://mlsanity.com/paper/2507.18996/) (2025)

## Abstract (arXiv)

This note continues study of exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the observations. Such processes can be used for detecting violations of the IID assumption, which is commonly made in machine learning. Violations of the IID assumption are sometimes referred to as dataset shift, and dataset shift is sometimes subdivided into concept shift, covariate shift, etc. Our primary interest is in concept shift, but we will also discuss exchangeability martingales that decompose perfectly into two components one of which detects concept shift and the other detects what we call label shift. Our methods will be based on techniques of conformal prediction.