mlsanityData to 7 Oct 2026

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

arXiv 2107.07511 ↗First posted 15 Jul 2021cs.LG cs.AI math.STcited by 777

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.

Key facts. Cited by 777 papers in the map (data through 7 Oct 2026). Builds on 41 of its 132 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

41 of 132 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2016Distribution-Free Predictive Inference For Regressioncited by 657
  5. 2019Conformalized Quantile Regressioncited by 620
  6. 2019Conformal Prediction Under Covariate Shiftcited by 511
  7. 2020Classification with Valid and Adaptive Coveragecited by 390
  8. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  9. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  10. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  11. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
  12. 2018CatBoost: gradient boosting with categorical features supportcited by 208
29 more
  1. 2019Predictive inference with the jackknife+cited by 206
  2. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 150
  3. 2021Testing for Outliers with Conformal p-valuescited by 133
  4. 2013Learning by Transductioncited by 120
  5. 2019Nested conformal prediction and quantile out-of-bag ensemble methodscited by 88
  6. 2013A Conformal Prediction Approach to Explore Functional Datacited by 85
  7. 2020Robust Validation: Confident Predictions Even When Distributions Shiftcited by 82
  8. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  9. 2018Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Datacited by 64
  10. 2019Flexible distribution-free conditional predictive bands using density estimatorscited by 64
  11. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  12. 2021Conformalized Survival Analysiscited by 53
  13. 2021Improving Conditional Coverage via Orthogonal Quantile Regressioncited by 48
  14. 2019Prediction and outlier detection in classification problemscited by 44
  15. 2021Distribution-free calibration guarantees for histogram binning without sample splittingcited by 44
  16. 2019With Malice Towards None: Assessing Uncertainty via Equalized Coveragecited by 43
  17. 2017An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controlscited by 36
  18. 2018Cautious Deep Learningcited by 35
  19. 2019Testing randomnesscited by 32
  20. 2019Conformal prediction with localizationcited by 22
  21. 2013Distribution-Free Distribution Regressioncited by 21
  22. 2020PAC Confidence Predictions for Deep Neural Network Classifierscited by 21
  23. 2017Fast Exact Conformalization of Lasso using Piecewise Linear Homotopycited by 20
  24. 2021Bayes-optimal prediction with frequentist coverage controlcited by 20
  25. 2019Computing Full Conformal Prediction Set with Approximate Homotopycited by 19
  26. 2017Inductive Conformal Martingales for Change-Point Detectioncited by 17
  27. 2021Private Prediction Setscited by 16
  28. 2021Nested Conformal Prediction Sets for Classification with Applications to Probation Datacited by 12
  29. 2021Root-finding Approaches for Computing Conformal Prediction Setcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 53; 2023: 111; 2024: 158; 2025: 184; 2026: 265.

777 papers.

Sort:

By area:

  1. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  2. 2022Confident Adaptive Language Modelingcited by 297
  3. 2022Conformal Risk Controlcited by 290
  4. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  5. 2023Conformal Language Modelingcited by 163
  6. 2022Adaptive Conformal Predictions for Time Seriescited by 155
  7. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  8. 2024Language Models with Conformal Factuality Guaranteescited by 128
  9. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  10. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  11. 2023Class-Conditional Conformal Prediction with Many Classescited by 109
  12. 2022Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imagingcited by 107
38 more
  1. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  2. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  3. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  4. 2023Improved Online Conformal Prediction via Strongly Adaptive Online Learningcited by 76
  5. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  6. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  7. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  8. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 71
  9. 2022Sequential Predictive Conformal Inference for Time Seriescited by 71
  10. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
  11. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  12. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  13. 2022Batch Multivalid Conformal Predictioncited by 66
  14. 2022Practical Adversarial Multivalid Conformal Predictioncited by 65
  15. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  16. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  17. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62
  18. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  19. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  20. 2022Conformal Prediction Under Feedback Covariate Shift for Biomolecular Designcited by 53
  21. 2023Conformal Prediction for Deep Classifier via Label Rankingcited by 52
  22. 2023How to Trust Your Diffusion Model: A Convex Optimization Approach to Conformal Risk Controlcited by 50
  23. 2024How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunitiescited by 50
  24. 2021Calibrated Multiple-Output Quantile Regression with Representation Learningcited by 49
  25. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  26. 2023Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictionscited by 46
  27. 2023High-Dimensional Prediction for Sequential Decision Makingcited by 41
  28. 2023Conformal Prediction for Time Series with Modern Hopfield Networkscited by 40
  29. 2022Achieving Risk Control in Online Learning Settingscited by 38
  30. 2022Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Setscited by 38
  31. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  32. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  33. 2022Predictive Inference with Feature Conformal Predictioncited by 33
  34. 2022Conffusion: Confidence Intervals for Diffusion Modelscited by 33
  35. 2023Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagationcited by 33
  36. 2023Conformal Prediction Regions for Time Series using Linear Complementarity Programmingcited by 33
  37. 2023Conformal Nucleus Samplingcited by 32
  38. 2023Conformal prediction under ambiguous ground truthcited by 31

50 most cited of 777 All 777 (JSON)