mlsanityData to 14 Oct 2026

Classifier Calibration: A survey on how to assess and improve predicted class probabilities

arXiv 2112.10327 ↗First posted 20 Dec 2021cs.LG stat.MLcited by 73

Not on the map. Why?

Abstract (arXiv)

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 37 of its 95 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

37 of 95 references.

References · 37

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  4. 2019When Does Label Smoothing Help?cited by 988
  5. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  6. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  7. 2019Conformalized Quantile Regressioncited by 629
  8. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  9. 2017On Fairness and Calibrationcited by 406
  10. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  11. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  12. 2019Verified Uncertainty Calibrationcited by 309
25 more
  1. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  2. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  3. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  4. 2019Evaluating model calibration in classificationcited by 168
  5. 2021Assessing Generalization of SGD via Disagreementcited by 117
  6. 2021Predicting with Confidence on Unseen Distributionscited by 116
  7. 2020Mitigating Bias in Calibration Error Estimationcited by 101
  8. 2020Local Temperature Scaling for Probability Calibrationcited by 95
  9. 2019Distribution Calibration for Regressioncited by 89
  10. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  11. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 78
  12. 2019Non-Parametric Calibration for Classificationcited by 57
  13. 2018Dirichlet-based Gaussian Processes for Large-scale Calibrated Classificationcited by 51
  14. 2019Accurate Uncertainty Estimation and Decomposition in Ensemble Learningcited by 42
  15. 2020Calibrated Reliable Regression using Maximum Mean Discrepancycited by 36
  16. 2021Meta-Cal: Well-controlled Post-hoc Calibration by Rankingcited by 33
  17. 2021Calibrated and Sharp Uncertainties in Deep Learning via Density Estimationcited by 33
  18. 2017Fast calibrated additive quantile regressioncited by 27
  19. 2020Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binningcited by 27
  20. 2015Binary Classifier Calibration using an Ensemble of Near Isotonic Regression Modelscited by 21
  21. 2018Loss-Calibrated Approximate Inference in Bayesian Neural Networkscited by 18
  22. 2021Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inferencecited by 16
  23. 2018Calibrating Deep Convolutional Gaussian Processescited by 13
  24. 2019Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Biascited by 11
  25. 2018Probability Calibration Treescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 20; 2025: 26; 2026: 22.

73 papers.

Sort:

By area:

  1. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  2. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  3. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 29
  4. 2023Spuriosity Didn't Kill the Classifier: Using Invariant Predictions to Harness Spurious Featurescited by 21
  5. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 17
  6. 2024Confidence Calibration of Classifiers with Many Classescited by 11
  7. 2023Evaluating Probabilistic Classifiers: The Triptychcited by 8
  8. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  9. 2025Generalized Venn and Venn-Abers Calibration with Applications in Conformal Predictioncited by 7
  10. 2025Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge resultscited by 7
  11. 2025MCGrad: Multicalibration at Web Scalecited by 5
  12. 2024Schroedinger's Threshold: When the AUC doesn't predict Accuracycited by 4
38 more
  1. 2024Improving Calibration by Relating Focal Loss, Temperature Scaling, and Propernesscited by 4
  2. 2024Accuracy-Preserving Calibration via Statistical Modeling on Probability Simplexcited by 3
  3. 2024Ordinal-Aware Calibration for Ordinal Classificationcited by 3
  4. 2025Random Forest Calibrationcited by 3
  5. 2025Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibrationcited by 3
  6. 2025Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniquescited by 3
  7. 2026Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibrationcited by 3
  8. 2024Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posteriorcited by 2
  9. 2024Calibrating Where It Matters: Constrained Temperature Scalingcited by 2
  10. 2024Selecting a classification performance measure: matching the measure to the problemcited by 2
  11. 2025An Entropic Metric for Measuring Calibration of Machine Learning Modelscited by 2
  12. 2025Sparse Activations as Conformal Predictorscited by 2
  13. 2025On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shiftcited by 2
  14. 2025Multiclass Local Calibration with the Jensen-Shannon Distancecited by 2
  15. 2026Hierarchies of Calibration: Classification meets Regressioncited by 2
  16. 2023Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectivescited by 1
  17. 2024Model-Free Local Recalibration of Neural Networkscited by 1
  18. 2024Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensorscited by 1
  19. 2024Block-Fused Attention-Driven Adaptively-Pooled ResNet Model for Improved Cervical Cancer Classificationcited by 1
  20. 2024On the calibration of powerset speaker diarization modelscited by 1
  21. 2024Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modelingcited by 1
  22. 2025Exploring the Impact of Explainable AI and Cognitive Capabilities on Users' Decisionscited by 1
  23. 2025Recalibrating binary probabilistic classifierscited by 1
  24. 2025On Equivariant Model Selection through the Lens of Uncertaintycited by 1
  25. 2025Estimating prevalence with precision and accuracycited by 1
  26. 2025Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantificationcited by 1
  27. 2025Classification errors distort findings in automated speech processing: examples and solutions from child-development researchcited by 1
  28. 2026Extending confidence calibration to generalised measures of variationcited by 1
  29. 2026The Well-Tempered Classifier: Some Elementary Properties of Temperature Scalingcited by 1
  30. 2026Recipes for Calibration Checks in Safety-Critical Applicationscited by 1
  31. 2023Positive Unlabeled Learning Selected Not At Random (PULSNAR): class proportion estimation when the SCAR assumption does not holdcited by 0
  32. 2024On the Stability of a non-hyperbolic nonlinear map with non-bounded set of non-isolated fixed points with applications to Machine Learningcited by 0
  33. 2024Robust quantum dots charge autotuning using neural network uncertaintycited by 0
  34. 2024Probabilistic Scoring Lists for Interpretable Machine Learningcited by 0
  35. 2024Trimming the Risk: Towards Reliable Continuous Training for Deep Learning Inspection Systemscited by 0
  36. 2025PandaSkill - Player Performance and Skill Rating in Esports: Application to League of Legendscited by 0
  37. 2025Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanationscited by 0
  38. 2025The distribution of calibrated likelihood functions on the probability-likelihood Aitchison simplexcited by 0

50 most cited of 73 All 73 (JSON)