# Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)

arXiv: https://arxiv.org/abs/2405.06627 · HTML: https://mlsanity.com/paper/2405.06627/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 10 May 2024.

## Builds on (most cited first, 26 of 26)

- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Adaptive Conformal Predictions for Time Series](https://mlsanity.com/paper/2202.07282/) (2022)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [Conformal Prediction With Conditional Guarantees](https://mlsanity.com/paper/2305.12616/) (2023)
- [Conformal Inference for Online Prediction with Arbitrary Distribution Shifts](https://mlsanity.com/paper/2208.08401/) (2022)
- [Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders](https://mlsanity.com/paper/2203.12742/) (2022)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Conformal PID Control for Time Series Prediction](https://mlsanity.com/paper/2307.16895/) (2023)
- [Improved Online Conformal Prediction via Strongly Adaptive Online Learning](https://mlsanity.com/paper/2302.07869/) (2023)
- [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Adaptive Conformal Prediction for Motion Planning among Dynamic Agents](https://mlsanity.com/paper/2212.00278/) (2022)
- [Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions](https://mlsanity.com/paper/2310.05921/) (2023)
- [Doubly Robust Calibration of Prediction Sets under Covariate Shift](https://mlsanity.com/paper/2203.01761/) (2022)
- [Achieving Risk Control in Online Learning Settings](https://mlsanity.com/paper/2205.09095/) (2022)
- [Non-Exchangeable Conformal Risk Control](https://mlsanity.com/paper/2310.01262/) (2023)
- [Bayesian Optimization with Conformal Prediction Sets](https://mlsanity.com/paper/2210.12496/) (2022)
- [JAWS: Auditing Predictive Uncertainty Under Covariate Shift](https://mlsanity.com/paper/2207.10716/) (2022)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [Bayesian Optimization with Formal Safety Guarantees via Online Conformal Prediction](https://mlsanity.com/paper/2306.17815/) (2023)
- [Randomization Tests for Adaptively Collected Data](https://mlsanity.com/paper/2301.05365/) (2023)
- [Risk-Controlling Model Selection via Guided Bayesian Optimization](https://mlsanity.com/paper/2312.01692/) (2023)

## Built on by (most cited first, 26 of 26)

- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Weighted Aggregation of Conformity Scores for Classification](https://mlsanity.com/paper/2407.10230/) (2024)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [Predictive inference for time series: why is split conformal effective despite temporal dependence?](https://mlsanity.com/paper/2510.02471/) (2025)
- [Conformal Prediction and Human Decision Making](https://mlsanity.com/paper/2503.11709/) (2025)
- [Conformal Risk Control for Non-Monotonic Losses](https://mlsanity.com/paper/2602.20151/) (2026)
- [What If We Had Used a Different App? Reliable Counterfactual KPI Analysis in Wireless Systems](https://mlsanity.com/paper/2410.00150/) (2024)
- [Conformal Prediction as Bayesian Quadrature](https://mlsanity.com/paper/2502.13228/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Online Conformal Prediction with Efficiency Guarantees](https://mlsanity.com/paper/2507.02496/) (2025)
- [Leave a Window Out: Modifying the Jackknife for Predictive Inference in Time Series](https://mlsanity.com/paper/2605.30292/) (2026)
- [Automatic AI Model Selection for Wireless Systems: Online Learning via Digital Twinning](https://mlsanity.com/paper/2406.15819/) (2024)
- [Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMs](https://mlsanity.com/paper/2506.13593/) (2025)
- [Conformal Prediction in The Loop: A Feedback-Based Uncertainty Model for Trajectory Optimization](https://mlsanity.com/paper/2510.16376/) (2025)
- [Conformal Prediction Adaptive to Unknown Subpopulation Shifts](https://mlsanity.com/paper/2506.05583/) (2025)
- [CP2: Leveraging Geometry for Conformal Prediction via Canonicalization](https://mlsanity.com/paper/2506.16189/) (2025)
- [Single Trajectory Conformal Prediction](https://mlsanity.com/paper/2406.01570/) (2024)
- [Uncertainty-Aware Prediction of Parkinson's Disease Medication Needs: A Two-Stage Conformal Prediction Approach](https://mlsanity.com/paper/2508.10284/) (2025)
- [Conformal Language Modeling via Posterior Sampling](https://mlsanity.com/paper/2606.03731/) (2026)
- [Relevance-Aware Thresholding in Online Conformal Prediction for Time Series](https://mlsanity.com/paper/2510.02809/) (2025)
- [Calibrated Uncertainty Sampling for Active Learning](https://mlsanity.com/paper/2510.03162/) (2025)
- [Weighted Conformal Prediction Provides Adaptive and Valid Mask-Conditional Coverage for General Missing Data Mechanisms](https://mlsanity.com/paper/2512.14221/) (2025)
- [The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift](https://mlsanity.com/paper/2607.18088/) (2026)
- [When Can Fraud Operations Authorize Automation? A Decision-Support Framework for Fresh Audit Evidence and Review Workload](https://mlsanity.com/paper/2608.08577/) (2026)
- [GenCAR: Generative Counterfactual Alignment with Risk-Controlled Selection for Out-of-Distribution Recommendation](https://mlsanity.com/paper/2609.02162/) (2026)
- [GeoDose-CP: Graph-Local Conformal Inference for Continuous-Treatment Earth Observation](https://mlsanity.com/paper/2609.28895/) (2026)

## Abstract (arXiv)

As artificial intelligence (AI) / machine learning (ML) gain widespread adoption, practitioners are increasingly seeking means to quantify and control the risk these systems incur. This challenge is especially salient when such systems have autonomy to collect their own data, such as in black-box optimization and active learning, where their actions induce sequential feedback-loop shifts in the data distribution. Conformal prediction is a promising approach to uncertainty and risk quantification, but prior variants' validity guarantees have assumed some form of ``quasi-exchangeability'' on the data distribution, thereby excluding many types of sequential shifts. In this paper we prove that conformal prediction can theoretically be extended to textitany joint data distribution, not just exchangeable or quasi-exchangeable ones. Although the most general case is exceedingly impractical to compute, for concrete practical applications we outline a procedure for deriving specific conformal algorithms for any data distribution, and we use this procedure to derive tractable algorithms for a series of AI/ML-agent-induced covariate shifts. We evaluate the proposed algorithms empirically on synthetic black-box optimization and active learning tasks.