# How to Evaluate Uncertainty Estimates in Machine Learning for Regression?

arXiv: https://arxiv.org/abs/2106.03395 · HTML: https://mlsanity.com/paper/2106.03395/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Evaluating time series forecasting models: An empirical study on performance estimation methods](https://mlsanity.com/paper/1905.11744/) (2019)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Approximation Trees: Statistical Stability in Model Distillation](https://mlsanity.com/paper/1808.07573/) (2018)
- [Tight Prediction Intervals Using Expanded Interval Minimization](https://mlsanity.com/paper/1806.11222/) (2018)
- [Prediction intervals for Deep Neural Networks](https://mlsanity.com/paper/2010.04044/) (2020)

## Built on by (most cited first, 15 of 15)

- [On the role of Model Uncertainties in Bayesian Optimization](https://mlsanity.com/paper/2301.05983/) (2023)
- [Uncertainty-Aware Online Extrinsic Calibration: A Conformal Prediction Approach](https://mlsanity.com/paper/2501.06878/) (2025)
- [Confident Neural Network Regression with Bootstrapped Deep Ensembles](https://mlsanity.com/paper/2202.10903/) (2022)
- [Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimates](https://mlsanity.com/paper/2303.08587/) (2023)
- [Composite Quantile Regression With XGBoost Using the Novel Arctan Pinball Loss](https://mlsanity.com/paper/2406.02293/) (2024)
- [Deploying scalable traffic prediction models for efficient management in real-world large transportation networks during hurricane evacuations](https://mlsanity.com/paper/2406.12119/) (2024)
- [Uncertainty Quantification of Wind Gust Predictions in the Northeast United States: An Evidential Neural Network and Explainable Artificial Intelligence Approach](https://mlsanity.com/paper/2502.00300/) (2025)
- [Multi-site modelling and reconstruction of past extreme skew surges along the French Atlantic coast](https://mlsanity.com/paper/2505.00835/) (2025)
- [Single- to multi-fidelity history-dependent learning with uncertainty quantification and disentanglement: application to data-driven constitutive modeling](https://mlsanity.com/paper/2507.13416/) (2025)
- [Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Models](https://mlsanity.com/paper/2508.17761/) (2025)
- [Multivariate Uncertainty Quantification with Tomographic Quantile Forests](https://mlsanity.com/paper/2512.16383/) (2025)
- [Taylor-Sensus Network: Embracing Noise to Enlighten Uncertainty for Scientific Data](https://mlsanity.com/paper/2409.07942/) (2024)
- [A Comprehensive Framework for Uncertainty Quantification of Voxel-wise Supervised Models in IVIM MRI](https://mlsanity.com/paper/2508.04588/) (2025)
- [A Physics-Guided Probabilistic Surrogate Modeling Framework for Digital Twins of Underwater Radiated Noise](https://mlsanity.com/paper/2509.25730/) (2025)
- [Filtering instances and rejecting predictions to obtain reliable models in healthcare](https://mlsanity.com/paper/2510.24368/) (2025)

## Abstract (arXiv)

As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Secondly, a better loglikelihood does not guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why it is fundamentally flawed to test a prediction or confidence interval on a single test set. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favoring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods.