# The Peril of Popular Deep Learning Uncertainty Estimation Methods

arXiv: https://arxiv.org/abs/2112.05000 · HTML: https://mlsanity.com/paper/2112.05000/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty](https://mlsanity.com/paper/2102.11409/) (2021)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)
- [Hamiltonian Monte Carlo for Hierarchical Models](https://mlsanity.com/paper/1312.0906/) (2013)
- [Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes](https://mlsanity.com/paper/2005.08140/) (2020)
- [Uncertainty Estimation Using a Single Deep Deterministic Neural Network](https://mlsanity.com/paper/2003.02037/) (2020)

## Built on by (most cited first, 12 of 12)

- [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022)
- [Layer Ensembles: A Single-Pass Uncertainty Estimation in Deep Learning for Segmentation](https://mlsanity.com/paper/2203.08878/) (2022)
- [Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity](https://mlsanity.com/paper/2309.06240/) (2023)
- [Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification](https://mlsanity.com/paper/2304.04042/) (2023)
- [Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts](https://mlsanity.com/paper/2303.07170/) (2023)
- [TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation](https://mlsanity.com/paper/2304.07728/) (2023)
- [Improving Generalization via Uncertainty Driven Perturbations](https://mlsanity.com/paper/2202.05737/) (2022)
- [Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models](https://mlsanity.com/paper/2508.11460/) (2025)
- [Multi-robot Multi-source Localization in Complex Flows with Physics-Preserving Environment Models](https://mlsanity.com/paper/2509.14228/) (2025)
- [Can input reconstruction be used to directly estimate uncertainty of a regression U-Net model? -- Application to proton therapy dose prediction for head and neck cancer patients](https://mlsanity.com/paper/2310.19686/) (2023)
- [From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network](https://mlsanity.com/paper/2512.13111/) (2025)
- [A Cubing Strategy for Identifying Stable Hyperparameter Regions for Uncertainty Quantification in Spatial Deep Learning](https://mlsanity.com/paper/2605.16570/) (2026)

## Abstract (arXiv)

Uncertainty estimation (UE) techniques -- such as the Gaussian process (GP), Bayesian neural networks (BNN), Monte Carlo dropout (MCDropout) -- aim to improve the interpretability of machine learning models by assigning an estimated uncertainty value to each of their prediction outputs. However, since too high uncertainty estimates can have fatal consequences in practice, this paper analyzes the above techniques. Firstly, we show that GP methods always yield high uncertainty estimates on out of distribution (OOD) data. Secondly, we show on a 2D toy example that both BNNs and MCDropout do not give high uncertainty estimates on OOD samples. Finally, we show empirically that this pitfall of BNNs and MCDropout holds on real world datasets as well. Our insights (i) raise awareness for the more cautious use of currently popular UE methods in Deep Learning, (ii) encourage the development of UE methods that approximate GP-based methods -- instead of BNNs and MCDropout, and (iii) our empirical setups can be used for verifying the OOD performances of any other UE method. The source code is available at https://github.com/epfml/uncertainity-estimation.