# Better Uncertainty Calibration via Proper Scores for Classification and Beyond

arXiv: https://arxiv.org/abs/2203.07835 · HTML: https://mlsanity.com/paper/2203.07835/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 77 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Unconstrained Monotonic Neural Networks](https://mlsanity.com/paper/1908.05164/) (2019)
- [Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration](https://mlsanity.com/paper/2109.14285/) (2021)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [A Broad Study on the Transferability of Visual Representations with Contrastive Learning](https://mlsanity.com/paper/2103.13517/) (2021)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks](https://mlsanity.com/paper/2003.06820/) (2020)
- [Post-hoc Uncertainty Calibration for Domain Drift Scenarios](https://mlsanity.com/paper/2012.10988/) (2020)
- [Non-Parametric Calibration for Classification](https://mlsanity.com/paper/1906.04933/) (2019)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [Being Bayesian about Categorical Probability](https://mlsanity.com/paper/2002.07965/) (2020)
- [On the folded normal distribution](https://mlsanity.com/paper/1402.3559/) (2014)

## Built on by (most cited first, 30 of 65)

- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Dual Focal Loss for Calibration](https://mlsanity.com/paper/2305.13665/) (2023)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach](https://mlsanity.com/paper/2505.01997/) (2025)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2405.03425/) (2024)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [Analysis and Comparison of Classification Metrics](https://mlsanity.com/paper/2209.05355/) (2022)
- [Improving Neural Additive Models with Bayesian Principles](https://mlsanity.com/paper/2305.16905/) (2023)
- [On (assessing) the fairness of risk score models](https://mlsanity.com/paper/2302.08851/) (2023)
- [Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration](https://mlsanity.com/paper/2408.02841/) (2024)
- [Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors](https://mlsanity.com/paper/2312.08589/) (2023)
- [Minimum-Risk Recalibration of Classifiers](https://mlsanity.com/paper/2305.10886/) (2023)
- [Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping](https://mlsanity.com/paper/2306.04985/) (2023)
- [Information-theoretic Generalization Analysis for Expected Calibration Error](https://mlsanity.com/paper/2405.15709/) (2024)
- [Continuous Autoregressive Language Models](https://mlsanity.com/paper/2510.27688/) (2025)
- [Continuous Visual Autoregressive Generation via Score Maximization](https://mlsanity.com/paper/2505.07812/) (2025)
- [Towards unraveling calibration biases in medical image analysis](https://mlsanity.com/paper/2305.05101/) (2023)
- [Language Generation with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2405.18906/) (2024)
- [Reassessing How to Compare and Improve the Calibration of Machine Learning Models](https://mlsanity.com/paper/2406.04068/) (2024)
- [Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood](https://mlsanity.com/paper/2402.15978/) (2024)
- [A calibration test for evaluating set-based epistemic uncertainty representations](https://mlsanity.com/paper/2502.16299/) (2025)
- [Performance Metrics for Probabilistic Ordinal Classifiers](https://mlsanity.com/paper/2309.08701/) (2023)
- [SAUC: Sparsity-Aware Uncertainty Calibration for Spatiotemporal Prediction with Graph Neural Networks](https://mlsanity.com/paper/2409.08766/) (2024)
- [Probabilistic Self-supervised Learning via Scoring Rules Minimization](https://mlsanity.com/paper/2309.02048/) (2023)
- [PAC-Bayes Analysis for Recalibration in Classification](https://mlsanity.com/paper/2406.06227/) (2024)
- [Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance](https://mlsanity.com/paper/2408.13648/) (2024)
- [GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks](https://mlsanity.com/paper/2410.09570/) (2024)

## Abstract (arXiv)

With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks. Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent. In this work, we introduce the framework of proper calibration errors, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties. This relationship can be used to reliably quantify the model calibration improvement. We theoretically and empirically demonstrate the shortcomings of commonly used estimators compared to our approach. Due to the wide applicability of proper scores, this gives a natural extension of recalibration beyond classification.