# On Calibrated Model Uncertainty in Deep Learning

arXiv: https://arxiv.org/abs/2206.07795 · HTML: https://mlsanity.com/paper/2206.07795/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [COVID-19 Image Data Collection](https://mlsanity.com/paper/2003.11597/) (2020)
- [Non-Parametric Calibration for Classification](https://mlsanity.com/paper/1906.04933/) (2019)
- [Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference](https://mlsanity.com/paper/1909.13550/) (2019)
- [Lung Infection Quantification of COVID-19 in CT Images with Deep Learning](https://mlsanity.com/paper/2003.04655/) (2020)
- [Loss-Calibrated Approximate Inference in Bayesian Neural Networks](https://mlsanity.com/paper/1805.03901/) (2018)
- [Bayesian Modelling in Practice: Using Uncertainty to Improve Trustworthiness in Medical Applications](https://mlsanity.com/paper/1906.08619/) (2019)

## Built on by (most cited first, 7 of 7)

- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Rethinking Data Distillation: Do Not Overlook Calibration](https://mlsanity.com/paper/2307.12463/) (2023)
- [Instant Uncertainty Calibration of NeRFs Using a Meta-Calibrator](https://mlsanity.com/paper/2312.02350/) (2023)
- [Improving Uncertainty-Error Correspondence in Deep Bayesian Medical Image Segmentation](https://mlsanity.com/paper/2409.03470/) (2024)
- [Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.14905/) (2024)
- [Validation of Conformal Prediction in Cervical Atypia Classification](https://mlsanity.com/paper/2505.08845/) (2025)
- [Uncertainty-Aware Vision-based Risk Object Identification via Conformal Risk Tube Prediction](https://mlsanity.com/paper/2603.23919/) (2026)

## Abstract (arXiv)

Estimated uncertainty by approximate posteriors in Bayesian neural networks are prone to miscalibration, which leads to overconfident predictions in critical tasks that have a clear asymmetric cost or significant losses. Here, we extend the approximate inference for the loss-calibrated Bayesian framework to dropweights based Bayesian neural networks by maximising expected utility over a model posterior to calibrate uncertainty in deep learning. Furthermore, we show that decisions informed by loss-calibrated uncertainty can improve diagnostic performance to a greater extent than straightforward alternatives. We propose Maximum Uncertainty Calibration Error (MUCE) as a metric to measure calibrated confidence, in addition to its prediction especially for high-risk applications, where the goal is to minimise the worst-case deviation between error and estimated uncertainty. In experiments, we show the correlation between error in prediction and estimated uncertainty by interpreting Wasserstein distance as the accuracy of prediction. We evaluated the effectiveness of our approach to detecting Covid-19 from X-Ray images. Experimental results show that our method reduces miscalibration considerably, without impacting the models accuracy and improves reliability of computer-based diagnostics.