# Structure and Distribution Metric for Quantifying the Quality of Uncertainty: Assessing Gaussian Processes, Deep Neural Nets, and Deep Neural Operators for Regression

arXiv: https://arxiv.org/abs/2203.04515 · HTML: https://mlsanity.com/paper/2203.04515/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Quality of Uncertainty Quantification for Bayesian Neural Network Inference](https://mlsanity.com/paper/1906.09686/) (2019)
- [A sequential sampling strategy for extreme event statistics in nonlinear dynamical systems](https://mlsanity.com/paper/1804.07240/) (2018)
- [Tight Prediction Intervals Using Expanded Interval Minimization](https://mlsanity.com/paper/1806.11222/) (2018)

## Built on by (most cited first, 5 of 5)

- [NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators](https://mlsanity.com/paper/2208.11866/) (2022)
- [Discovering and forecasting extreme events via active learning in neural operators](https://mlsanity.com/paper/2204.02488/) (2022)
- [Improved generalization with deep neural operators for engineering systems: Path towards digital twin](https://mlsanity.com/paper/2301.06701/) (2023)
- [How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?](https://mlsanity.com/paper/2302.03679/) (2023)
- [Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier models](https://mlsanity.com/paper/2208.13080/) (2022)

## Abstract (arXiv)

We propose two bounded comparison metrics that may be implemented to arbitrary dimensions in regression tasks. One quantifies the structure of uncertainty and the other quantifies the distribution of uncertainty. The structure metric assesses the similarity in shape and location of uncertainty with the true error, while the distribution metric quantifies the supported magnitudes between the two. We apply these metrics to Gaussian Processes (GPs), Ensemble Deep Neural Nets (DNNs), and Ensemble Deep Neural Operators (DNOs) on high-dimensional and nonlinear test cases. We find that comparing a model's uncertainty estimates with the model's squared error provides a compelling ground truth assessment. We also observe that both DNNs and DNOs, especially when compared to GPs, provide encouraging metric values in high dimensions with either sparse or plentiful data.