# Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

arXiv: https://arxiv.org/abs/1910.12656 · HTML: https://mlsanity.com/paper/1910.12656/

## Key facts

- Cited by 364 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2019.

## Built on by (most cited first, 30 of 364)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration](https://mlsanity.com/paper/2109.14285/) (2021)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [Uncertainty Quantification over Graph with Conformalized Graph Neural Networks](https://mlsanity.com/paper/2305.14535/) (2023)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing](https://mlsanity.com/paper/2309.12236/) (2023)
- [Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization](https://mlsanity.com/paper/2102.02414/) (2021)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration](https://mlsanity.com/paper/2203.13834/) (2022)
- [Calibration and Correctness of Language Models for Code](https://mlsanity.com/paper/2402.02047/) (2024)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [Low-Degree Multicalibration](https://mlsanity.com/paper/2203.01255/) (2022)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Uncertainty in Language Models: Assessment through Rank-Calibration](https://mlsanity.com/paper/2404.03163/) (2024)
- [Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration](https://mlsanity.com/paper/2109.14591/) (2021)
- [What Makes Graph Neural Networks Miscalibrated?](https://mlsanity.com/paper/2210.06391/) (2022)

## Abstract (arXiv)

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.