mlsanityData to 7 Oct 2026

Distribution-Free, Risk-Controlling Prediction Sets

arXiv 2101.02703 ↗First posted 7 Jan 2021cs.LG cs.AI cs.CVcited by 278

Not on the map. Why?

Abstract (arXiv)

While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making. Deploying learning systems in consequential settings also requires calibrating and communicating the uncertainty of predictions. To convey instance-wise uncertainty for prediction tasks, we show how to generate set-valued predictions from a black-box predictor that control the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets. This framework enables simple, distribution-free, rigorous error control for many tasks, and we demonstrate it in five large-scale machine learning problems: (1) classification problems where some mistakes are more costly than others; (2) multi-label classification, where each observation has multiple associated labels; (3) classification problems where the labels have a hierarchical structure; (4) image segmentation, where we wish to predict a set of pixels containing an object of interest; and (5) protein structure prediction. Lastly, we discuss extensions to uncertainty quantification for ranking, metric learning and distributionally robust learning.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Builds on 23 of its 63 references that are in the map. First posted on arXiv 7 Jan 2021.

Builds on

23 of 63 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2019Conformalized Quantile Regressioncited by 620
  5. 2019Conformal Prediction Under Covariate Shiftcited by 511
  6. 2020PraNet: Parallel Reverse Attention Network for Polyp Segmentationcited by 460
  7. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  8. 2020Classification with Valid and Adaptive Coveragecited by 390
  9. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  10. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
  11. 2019Predictive inference with the jackknife+cited by 206
  12. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 150
11 more
  1. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  2. 2019Nested conformal prediction and quantile out-of-bag ensemble methodscited by 88
  3. 2013A Conformal Prediction Approach to Explore Functional Datacited by 85
  4. 2020Robust Validation: Confident Predictions Even When Distributions Shiftcited by 82
  5. 2019Flexible distribution-free conditional predictive bands using density estimatorscited by 64
  6. 2019PAC Confidence Sets for Deep Neural Networks via Calibrated Predictioncited by 59
  7. 2019Prediction and outlier detection in classification problemscited by 44
  8. 2018Cautious Deep Learningcited by 35
  9. 2019Conformal calibratorscited by 22
  10. 2019Conformal prediction with localizationcited by 22
  11. 2020PAC Confidence Predictions for Deep Neural Network Classifierscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 37; 2023: 34; 2024: 43; 2025: 47; 2026: 101.

278 papers.

Sort:

By area:

  1. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  2. 2022Confident Adaptive Language Modelingcited by 297
  3. 2022Conformal Risk Controlcited by 290
  4. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  5. 2023Conformal Language Modelingcited by 163
  6. 2021Testing for Outliers with Conformal p-valuescited by 133
  7. 2021Learning Optimal Conformal Classifierscited by 108
  8. 2022Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imagingcited by 107
  9. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  10. 2023Conformal PID Control for Time Series Predictioncited by 85
  11. 2023Uncertainty Quantification over Graph with Conformalized Graph Neural Networkscited by 79
  12. 2023Improved Online Conformal Prediction via Strongly Adaptive Online Learningcited by 76
38 more
  1. 2021Fair Conformal Predictors for Applications in Medical Imagingcited by 74
  2. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  3. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  4. 2022Selection by Prediction with Conformal p-valuescited by 66
  5. 2022Training Uncertainty-Aware Classifiers with Conformalized Deep Learningcited by 62
  6. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  7. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  8. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  9. 2023Conformal Prediction for Deep Classifier via Label Rankingcited by 52
  10. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  11. 2023How to Trust Your Diffusion Model: A Convex Optimization Approach to Conformal Risk Controlcited by 50
  12. 2021Calibrated Multiple-Output Quantile Regression with Representation Learningcited by 49
  13. 2021Improving Conditional Coverage via Orthogonal Quantile Regressioncited by 48
  14. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  15. 2021PAC Prediction Sets Under Covariate Shiftcited by 42
  16. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39
  17. 2021Tracking the risk of a deployed model and detecting harmful distribution shiftscited by 38
  18. 2022Achieving Risk Control in Online Learning Settingscited by 38
  19. 2022Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Setscited by 38
  20. 2022Predictive Inference with Feature Conformal Predictioncited by 33
  21. 2022Conffusion: Confidence Intervals for Diffusion Modelscited by 33
  22. 2023Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagationcited by 33
  23. 2022Calibrated Selective Classificationcited by 30
  24. 2023Non-Exchangeable Conformal Risk Controlcited by 29
  25. 2022Conformal Prediction Sets with Limited False Positivescited by 28
  26. 2023Principal Uncertainty Quantification with Spatial Correlation for Image Restoration Problemscited by 28
  27. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  28. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  29. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  30. 2022Efficient and Differentiable Conformal Prediction with General Function Classescited by 25
  31. 2022Conformal Prediction with Temporal Quantile Adjustmentscited by 25
  32. 2024Calibrated Uncertainty Quantification for Operator Learning via Conformal Predictioncited by 25
  33. 2024Conformal Prediction: A Data Perspectivecited by 25
  34. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  35. 2023TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Predictioncited by 23
  36. 2024Adaptive Bounding Box Uncertainties via Two-Step Conformal Predictioncited by 23
  37. 2022Semantic uncertainty intervals for disentangled latent spacescited by 22
  38. 2023Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signalingcited by 22

50 most cited of 278 All 278 (JSON)