# Uncertainty Quantification Metrics for Deep Regression

arXiv: https://arxiv.org/abs/2405.04278 · HTML: https://mlsanity.com/paper/2405.04278/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End](https://mlsanity.com/paper/2006.03349/) (2020)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Wasserstein Distances for Stereo Disparity Estimation](https://mlsanity.com/paper/2007.03085/) (2020)
- [Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles](https://mlsanity.com/paper/2305.01754/) (2023)
- [ProbFlow: Joint Optical Flow and Uncertainty Estimation](https://mlsanity.com/paper/1708.06509/) (2017)
- [Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel](https://mlsanity.com/paper/1906.00588/) (2019)
- [Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation: A Benchmark Study](https://mlsanity.com/paper/2012.15772/) (2020)
- [NOMU: Neural Optimization-based Model Uncertainty](https://mlsanity.com/paper/2102.13640/) (2021)
- [Learning Sample Difficulty from Pre-trained Models for Reliable Prediction](https://mlsanity.com/paper/2304.10127/) (2023)
- [Energy-Based Models for Deep Probabilistic Regression](https://mlsanity.com/paper/1909.12297/) (2019)

## Built on by (most cited first, 16 of 16)

- [Continuous Normalizing Flows for Uncertainty-Aware Human Pose Estimation](https://mlsanity.com/paper/2505.02287/) (2025)
- [CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment](https://mlsanity.com/paper/2505.22803/) (2025)
- [Probabilistic Neural Networks (PNNs) with t-Distributed Outputs: Adaptive Prediction Intervals Beyond Gaussian Assumptions](https://mlsanity.com/paper/2503.12354/) (2025)
- [Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Models](https://mlsanity.com/paper/2508.17761/) (2025)
- [Principled Confidence Estimation for Deep Computed Tomography](https://mlsanity.com/paper/2602.05812/) (2026)
- [Laplace Approximation for Bayesian Tensor Network Kernel Machines](https://mlsanity.com/paper/2604.26673/) (2026)
- [TRUST: Test-time Resource Utilization for Superior Trustworthiness](https://mlsanity.com/paper/2506.06048/) (2025)
- [MATTER: Multiscale Attention for Registration Error Regression](https://mlsanity.com/paper/2509.12924/) (2025)
- [ViscNet: Vision-Based In-line Viscometry for Fluid Mixing Process](https://mlsanity.com/paper/2512.01268/) (2025)
- [R^2-HGP: A Double-Regularized Gaussian Process for Heterogeneous Transfer Learning](https://mlsanity.com/paper/2512.10258/) (2025)
- [GeoBlur: Epipolar Geometry Estimation from a Single Motion-Blurred Image](https://mlsanity.com/paper/2605.01552/) (2026)
- [Neuromorphic Monocular Depth Estimation with Uncertainty Modeling](https://mlsanity.com/paper/2605.10675/) (2026)
- [Uncertainty-Calibrated Recommendations for Low-Active Users](https://mlsanity.com/paper/2605.17788/) (2026)
- [Error Aware Distribution Prediction for Lightweight Implicit Neural Representations](https://mlsanity.com/paper/2607.10068/) (2026)
- [Gaussian-Mixture Latent Flow for Stochastic 3D Human Motion Prediction](https://mlsanity.com/paper/2608.21093/) (2026)
- [A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction Models](https://mlsanity.com/paper/2608.29705/) (2026)

## Abstract (arXiv)

When deploying deep neural networks on robots or other physical systems, the learned model should reliably quantify predictive uncertainty. A reliable uncertainty allows downstream modules to reason about the safety of its actions. In this work, we address metrics for evaluating such an uncertainty. Specifically, we focus on regression tasks, and investigate Area Under Sparsification Error (AUSE), Calibration Error, Spearman's Rank Correlation, and Negative Log-Likelihood (NLL). Using synthetic regression datasets, we look into how those metrics behave under four typical types of uncertainty, their stability regarding the size of the test set, and reveal their strengths and weaknesses. Our results indicate that Calibration Error is the most stable and interpretable metric, but AUSE and NLL also have their respective use cases. We discourage the usage of Spearman's Rank Correlation for evaluating uncertainties and recommend replacing it with AUSE.