# WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales

arXiv: https://arxiv.org/abs/2505.04608 · HTML: https://mlsanity.com/paper/2505.04608/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 87 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Theoretical Foundations of Conformal Prediction](https://mlsanity.com/paper/2411.11824/) (2024)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Universal Inference](https://mlsanity.com/paper/1912.11436/) (2019)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Doubly Robust Calibration of Prediction Sets under Covariate Shift](https://mlsanity.com/paper/2203.01761/) (2022)
- [Nonparametric Two-Sample Testing by Betting](https://mlsanity.com/paper/2112.09162/) (2021)
- [Testing randomness](https://mlsanity.com/paper/1906.09256/) (2019)
- [Testing exchangeability: fork-convexity, supermartingales, and e-processes](https://mlsanity.com/paper/2102.00630/) (2021)
- [Retrain or not retrain: Conformal test martingales for change-point detection](https://mlsanity.com/paper/2102.10439/) (2021)
- [E-detectors: a nonparametric framework for sequential change detection](https://mlsanity.com/paper/2203.03532/) (2022)
- [Model-free selective inference under covariate shift via weighted conformal p-values](https://mlsanity.com/paper/2307.09291/) (2023)
- [Non-Exchangeable Conformal Risk Control](https://mlsanity.com/paper/2310.01262/) (2023)
- [Bayesian Optimization with Conformal Prediction Sets](https://mlsanity.com/paper/2210.12496/) (2022)
- [Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)](https://mlsanity.com/paper/2405.06627/) (2024)
- [E-Values Expand the Scope of Conformal Prediction](https://mlsanity.com/paper/2503.13050/) (2025)
- [Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach](https://mlsanity.com/paper/2408.07511/) (2024)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [Inductive Conformal Martingales for Change-Point Detection](https://mlsanity.com/paper/1706.03415/) (2017)
- [Adapting to Continuous Covariate Shift via Online Density Ratio Estimation](https://mlsanity.com/paper/2302.02552/) (2023)

## Built on by (most cited first, 16 of 16)

- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [Monitoring Risks in Test-Time Adaptation](https://mlsanity.com/paper/2507.08721/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://mlsanity.com/paper/2602.13848/) (2026)
- [Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG](https://mlsanity.com/paper/2605.29139/) (2026)
- [Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers](https://mlsanity.com/paper/2606.11949/) (2026)
- [CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning](https://mlsanity.com/paper/2607.20129/) (2026)
- [Information-Geometric First-Passage Monitoring of Distributional Stability in Stochastic Systems](https://mlsanity.com/paper/2604.24083/) (2026)
- [Online Safety Monitoring for LLMs](https://mlsanity.com/paper/2607.02510/) (2026)
- [Calibrated e-CUSUM Decoding for Quantized Reasoning Models: Why Token Log-Probability Is the Wrong Observable for Decoding Monitors](https://mlsanity.com/paper/2607.11317/) (2026)
- [Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers](https://mlsanity.com/paper/2608.14089/) (2026)
- [What Changed? Drift Detection with Real, Virtual, and Incomparable Diagnosis](https://mlsanity.com/paper/2609.27865/) (2026)
- [Protected Cores Are Not Enough: Certifying AI-Proposed Revisions of Temporal Specifications](https://mlsanity.com/paper/2609.33461/) (2026)
- [The Premise Is the Problem: Exchangeability Failure in Self-Monitored Test-Time Adaptation](https://mlsanity.com/paper/2610.07038/) (2026)
- [Backward Conformal Prediction under Covariate Shift](https://mlsanity.com/paper/2610.16048/) (2026)
- [Monitoring for Shifts at Unknown Times: Parameter-Free Conditional Conformal Martingales](https://mlsanity.com/paper/2610.17343/) (2026)

## Abstract (arXiv)

Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continual, post-deployment monitoring to quickly detect and address any unsafe behavior. Methods for nonparametric sequential testing -- especially conformal test martingales (CTMs) and anytime-valid inference -- offer promising tools for this monitoring task. However, existing approaches are restricted to monitoring limited hypothesis classes or ``alarm criteria'' (e.g., detecting data shifts that violate certain exchangeability or IID assumptions), do not allow for online adaptation in response to shifts, and/or cannot diagnose the cause of degradation or alarm. In this paper, we address these limitations by proposing a weighted generalization of conformal test martingales (WCTMs), which lay a theoretical foundation for online monitoring for any unexpected changepoints in the data distribution while controlling false-alarms. For practical applications, we propose specific WCTM algorithms that adapt online to mild covariate shifts (in the marginal input distribution), quickly detect harmful shifts, and diagnose those harmful shifts as concept shifts (in the conditional label distribution) or extreme (out-of-support) covariate shifts that cannot be easily adapted to. On real-world datasets, we demonstrate improved performance relative to state-of-the-art baselines.