mlsanityData to 7 Oct 2026

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

arXiv 2110.01052 ↗First posted 3 Oct 2021cs.LG cs.AI cs.CVcited by 256

Not on the map. Why?

Abstract (arXiv)

We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use the framework to provide new calibration methods for several core machine learning tasks, with detailed worked examples in computer vision and tabular medical data.

Key facts. Cited by 256 papers in the map (data through 7 Oct 2026). Builds on 32 of its 84 references that are in the map. First posted on arXiv 3 Oct 2021.

Builds on

32 of 84 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Mask R-CNNcited by 7,259
  4. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  5. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  6. 2019Conformalized Quantile Regressioncited by 620
  7. 2019Conformal Prediction Under Covariate Shiftcited by 511
  8. 2020Classification with Valid and Adaptive Coveragecited by 390
  9. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  10. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  11. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  12. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
20 more
  1. 2019Predictive inference with the jackknife+cited by 206
  2. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 150
  3. 2021Testing for Outliers with Conformal p-valuescited by 133
  4. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  5. 2013A Conformal Prediction Approach to Explore Functional Datacited by 85
  6. 2020Robust Validation: Confident Predictions Even When Distributions Shiftcited by 82
  7. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  8. 2018Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Datacited by 64
  9. 2019Flexible distribution-free conditional predictive bands using density estimatorscited by 64
  10. 2019PAC Confidence Sets for Deep Neural Networks via Calibrated Predictioncited by 59
  11. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  12. 2021Conformalized Survival Analysiscited by 53
  13. 2019Prediction and outlier detection in classification problemscited by 44
  14. 2021PAC Prediction Sets Under Covariate Shiftcited by 42
  15. 2018Cautious Deep Learningcited by 35
  16. 2019Testing randomnesscited by 32
  17. 2019Conformal calibratorscited by 22
  18. 2019Conformal prediction with localizationcited by 22
  19. 2021Private Prediction Setscited by 16
  20. 2021Nested Conformal Prediction Sets for Classification with Applications to Probation Datacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 25; 2024: 44; 2025: 38; 2026: 129.

256 papers.

Sort:

By area:

  1. 2022Conformal Risk Controlcited by 290
  2. 2023Conformal Language Modelingcited by 163
  3. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  4. 2022Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imagingcited by 107
  5. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  6. 2024Large language model validity via enhanced conformal prediction methodscited by 91
  7. 2023Improved Online Conformal Prediction via Strongly Adaptive Online Learningcited by 76
  8. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
  9. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  10. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  11. 2022Conformal Prediction Under Feedback Covariate Shift for Biomolecular Designcited by 53
  12. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
38 more
  1. 2023How to Trust Your Diffusion Model: A Convex Optimization Approach to Conformal Risk Controlcited by 50
  2. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  3. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  4. 2022Achieving Risk Control in Online Learning Settingscited by 38
  5. 2022Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Setscited by 38
  6. 2025Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agentscited by 37
  7. 2022Predictive Inference with Feature Conformal Predictioncited by 33
  8. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  9. 2023Principal Uncertainty Quantification with Spatial Correlation for Image Restoration Problemscited by 28
  10. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  11. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  12. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  13. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  14. 2022T-Cal: An optimal test for the calibration of predictive modelscited by 24
  15. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  16. 2024Adaptive Bounding Box Uncertainties via Two-Step Conformal Predictioncited by 23
  17. 2023Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signalingcited by 22
  18. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
  19. 2024An Information Theoretic Perspective on Conformal Predictioncited by 21
  20. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  21. 2025Thought calibration: Efficient and confident test-time scalingcited by 20
  22. 2023Statistical Inference for Fairness Auditingcited by 19
  23. 2024Conformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertaintycited by 19
  24. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  25. 2022MAPIE: an open-source library for distribution-free uncertainty quantificationcited by 17
  26. 2023Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Modelscited by 17
  27. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  28. 2023Conformal Autoregressive Generation: Beam Search with Coverage Guaranteescited by 16
  29. 2024Prune 'n Predict: Optimizing LLM Decision-making with Conformal Predictioncited by 15
  30. 2022Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictionscited by 14
  31. 2023Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithmscited by 14
  32. 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 14
  33. 2022Recommendation Systems with Distribution-Free Reliability Guaranteescited by 13
  34. 2024Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selectioncited by 13
  35. 2024Semi-Supervised Risk Control via Prediction-Powered Inferencecited by 13
  36. 2022Posterior-Variance-Based Error Quantification for Inverse Problems in Imagingcited by 12
  37. 2024Does confidence calibration improve conformal prediction?cited by 12
  38. 2024Bellman Conformal Inference: Calibrating Prediction Intervals For Time Seriescited by 12

50 most cited of 256 All 256 (JSON)