# Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms

arXiv: https://arxiv.org/abs/2004.10710 · HTML: https://mlsanity.com/paper/2004.10710/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 30 of 34)

- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [Quantification of Uncertainty with Adversarial Models](https://mlsanity.com/paper/2307.03217/) (2023)
- [Deep Ensembles from a Bayesian Perspective](https://mlsanity.com/paper/2105.13283/) (2021)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment](https://mlsanity.com/paper/2101.08477/) (2021)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Uncertainty Quantification by Ensemble Learning for Computational Optical Form Measurements](https://mlsanity.com/paper/2103.01259/) (2021)
- [Empirical Validation of Conformal Prediction for Trustworthy Skin Lesions Classification](https://mlsanity.com/paper/2312.07460/) (2023)
- [Geometry-Informed Neural Operator Transformer](https://mlsanity.com/paper/2504.19452/) (2025)
- [Machine Learning and Cosmology](https://mlsanity.com/paper/2203.08056/) (2022)
- [Artificial intelligence approaches for materials-by-design of energetic materials: state-of-the-art, challenges, and future directions](https://mlsanity.com/paper/2211.08179/) (2022)
- [A General Approach for Determining Applicability Domain of Machine Learning Models](https://mlsanity.com/paper/2406.05143/) (2024)
- [The information of attribute uncertainties: what convolutional neural networks can learn about errors in input data](https://mlsanity.com/paper/2108.04742/) (2021)
- [A framework for benchmarking uncertainty in deep regression](https://mlsanity.com/paper/2109.09048/) (2021)
- [Fairness Uncertainty Quantification: How certain are you that the model is fair?](https://mlsanity.com/paper/2304.13950/) (2023)
- [Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making](https://mlsanity.com/paper/2505.18066/) (2025)
- [Sample-Efficient Optimisation with Probabilistic Transformer Surrogates](https://mlsanity.com/paper/2205.13902/) (2022)
- [A view on model misspecification in uncertainty quantification](https://mlsanity.com/paper/2210.16938/) (2022)
- [DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods](https://mlsanity.com/paper/2411.08587/) (2024)
- [Gated Linear Model induced U-net for surrogate modeling and uncertainty quantification](https://mlsanity.com/paper/2111.05123/) (2021)
- [Self-Normalized Density Map (SNDM) for Counting Microbiological Objects](https://mlsanity.com/paper/2203.09474/) (2022)
- [Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks](https://mlsanity.com/paper/2206.05643/) (2022)
- [Transferring climate change physical knowledge](https://mlsanity.com/paper/2309.14780/) (2023)
- [PULASki: Learning inter-rater variability using statistical distances to improve probabilistic segmentation](https://mlsanity.com/paper/2312.15686/) (2023)
- [Enhancing Suicide Risk Detection on Social Media through Semi-Supervised Deep Label Smoothing](https://mlsanity.com/paper/2405.05795/) (2024)
- [Legitimate ground-truth-free metrics for deep uncertainty classification scoring](https://mlsanity.com/paper/2410.23046/) (2024)
- [Multivariate Uncertainty Quantification with Tomographic Quantile Forests](https://mlsanity.com/paper/2512.16383/) (2025)
- [A Ground-Truth Framework for Uncertainty Disentanglement with Posterior Risk](https://mlsanity.com/paper/2608.05995/) (2026)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [On planetary systems as ordered sequences](https://mlsanity.com/paper/2105.09966/) (2021)

## Abstract (arXiv)

We present a comparison of methods for uncertainty quantification (UQ) in deep learning algorithms in the context of a simple physical system. Three of the most common uncertainty quantification methods - Bayesian Neural Networks (BNN), Concrete Dropout (CD), and Deep Ensembles (DE) - are compared to the standard analytic error propagation. We discuss this comparison in terms endemic to both machine learning ("epistemic" and "aleatoric") and the physical sciences ("statistical" and "systematic"). The comparisons are presented in terms of simulated experimental measurements of a single pendulum - a prototypical physical system for studying measurement and analysis techniques. Our results highlight some pitfalls that may occur when using these UQ methods. For example, when the variation of noise in the training set is small, all methods predicted the same relative uncertainty independently of the inputs. This issue is particularly hard to avoid in BNN. On the other hand, when the test set contains samples far from the training distribution, we found that no methods sufficiently increased the uncertainties associated to their predictions. This problem was particularly clear for CD. In light of these results, we make some recommendations for usage and interpretation of UQ methods.