# Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation

arXiv: https://arxiv.org/abs/2112.07184 · HTML: https://mlsanity.com/paper/2112.07184/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 18 of 18)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [Calibration of Encoder Decoder Models for Neural Machine Translation](https://mlsanity.com/paper/1903.00802/) (2019)
- [Calibrated Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1906.08312/) (2019)
- [Autoregressive Quantile Flows for Predictive Uncertainty Estimation](https://mlsanity.com/paper/2112.04643/) (2021)
- [On-the-Job Learning with Bayesian Decision Theory](https://mlsanity.com/paper/1506.03140/) (2015)

## Built on by (most cited first, 30 of 33)

- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [A Large-Scale Study of Probabilistic Calibration in Neural Network Regression](https://mlsanity.com/paper/2306.02738/) (2023)
- [Proximity-Informed Calibration for Deep Neural Networks](https://mlsanity.com/paper/2306.04590/) (2023)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts](https://mlsanity.com/paper/2302.06495/) (2023)
- [Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreement](https://mlsanity.com/paper/2305.01481/) (2023)
- [Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors](https://mlsanity.com/paper/2312.08589/) (2023)
- [On the Calibration of Human Pose Estimation](https://mlsanity.com/paper/2311.17105/) (2023)
- [On the role of Model Uncertainties in Bayesian Optimization](https://mlsanity.com/paper/2301.05983/) (2023)
- [Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shifts](https://mlsanity.com/paper/2403.05600/) (2024)
- [Probabilistic Calibration by Design for Neural Network Regression](https://mlsanity.com/paper/2403.11964/) (2024)
- [Sharp Calibrated Gaussian Processes](https://mlsanity.com/paper/2302.11961/) (2023)
- [Conformalizing Machine Translation Evaluation](https://mlsanity.com/paper/2306.06221/) (2023)
- [Uncertainty in Action: Confidence Elicitation in Embodied Agents](https://mlsanity.com/paper/2503.10628/) (2025)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Multi-Perspective Consistency Enhances Confidence Estimation in Large Language Models](https://mlsanity.com/paper/2402.11279/) (2024)
- [Calibrated Regression Against An Adversary Without Regret](https://mlsanity.com/paper/2302.12196/) (2023)
- [Calibrated and Conformal Propensity Scores for Causal Effect Estimation](https://mlsanity.com/paper/2306.00382/) (2023)
- [Accuracy-Preserving Calibration via Statistical Modeling on Probability Simplex](https://mlsanity.com/paper/2402.13765/) (2024)
- [Optimizing Estimators of Squared Calibration Errors in Classification](https://mlsanity.com/paper/2410.07014/) (2024)
- [Multivariate Latent Recalibration for Conditional Normalizing Flows](https://mlsanity.com/paper/2505.16636/) (2025)
- [Weighted Risk Invariance: Domain Generalization under Invariant Feature Shift](https://mlsanity.com/paper/2407.18428/) (2024)
- [Weakly Supervised Learners for Correction of AI Errors with Provable Performance Guarantees](https://mlsanity.com/paper/2402.00899/) (2024)
- [Model-Free Local Recalibration of Neural Networks](https://mlsanity.com/paper/2403.05756/) (2024)
- [Positional Encoder Graph Quantile Neural Networks for Geographic Data](https://mlsanity.com/paper/2409.18865/) (2024)
- [Hierarchical Uncertainty-Aware Graph Neural Network](https://mlsanity.com/paper/2504.19820/) (2025)
- [SphUnc: Hyperspherical Uncertainty Decomposition and Causal Identification via Information Geometry](https://mlsanity.com/paper/2603.01168/) (2026)
- [Calibrating Scientific Foundation Models with Inference-Time Stochastic Attention](https://mlsanity.com/paper/2604.19530/) (2026)
- [L2-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Error](https://mlsanity.com/paper/2510.13450/) (2025)

## Abstract (arXiv)

Accurate probabilistic predictions can be characterized by two properties -- calibration and sharpness. However, standard maximum likelihood training yields models that are poorly calibrated and thus inaccurate -- a 90% confidence interval typically does not contain the true outcome 90% of the time. This paper argues that calibration is important in practice and is easy to maintain by performing low-dimensional density estimation. We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance; unlike previous approaches, ours ensures the most general property of distribution calibration and applies to any model, including neural networks. We formally prove the correctness of our procedure assuming that we can estimate densities in low dimensions and we establish uniform convergence bounds. Our results yield empirical performance improvements on linear and deep Bayesian models and suggest that calibration should be increasingly leveraged across machine learning. We release a library that implements our methods along with a blog post here: https://shachideshpande.github.io/blog-distribution-calibration/.