# Evaluation of machine learning architectures on the quantification of epistemic and aleatoric uncertainties in complex dynamical systems

arXiv: https://arxiv.org/abs/2306.15159 · HTML: https://mlsanity.com/paper/2306.15159/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 60 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [Bayesian Neural Networks: An Introduction and Survey](https://mlsanity.com/paper/2006.12024/) (2020)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Manifold Gaussian Processes for Regression](https://mlsanity.com/paper/1402.5876/) (2014)
- [Convergence of Sparse Variational Inference in Gaussian Processes Regression](https://mlsanity.com/paper/2008.00323/) (2020)
- [Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks](https://mlsanity.com/paper/1707.02476/) (2017)
- [Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes](https://mlsanity.com/paper/2005.08140/) (2020)
- [NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators](https://mlsanity.com/paper/2208.11866/) (2022)
- [Discovering and forecasting extreme events via active learning in neural operators](https://mlsanity.com/paper/2204.02488/) (2022)
- [Last Layer Marginal Likelihood for Invariance Learning](https://mlsanity.com/paper/2106.07512/) (2021)
- [A sequential sampling strategy for extreme event statistics in nonlinear dynamical systems](https://mlsanity.com/paper/1804.07240/) (2018)
- [Output-Weighted Sampling for Multi-Armed Bandits with Extreme Payoffs](https://mlsanity.com/paper/2102.10085/) (2021)
- [Sequential Bayesian experimental design for estimation of extreme-event probability in stochastic dynamical systems](https://mlsanity.com/paper/2102.11108/) (2021)
- [Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier models](https://mlsanity.com/paper/2208.13080/) (2022)

## Built on by (most cited first, 2 of 2)

- [Active search for Bifurcations](https://mlsanity.com/paper/2406.11141/) (2024)
- [A generalized likelihood-weighted optimal sampling algorithm for rare-event probability quantification](https://mlsanity.com/paper/2310.14457/) (2023)

## Abstract (arXiv)

Machine learning methods for the construction of data-driven reduced order model models are used in an increasing variety of engineering domains, especially as a supplement to expensive computational fluid dynamics for design problems. An important check on the reliability of surrogate models is Uncertainty Quantification (UQ), a self assessed estimate of the model error. Accurate UQ allows for cost savings by reducing both the required size of training data sets and the required safety factors, while poor UQ prevents users from confidently relying on model predictions. We examine several machine learning techniques, including both Gaussian processes and a family UQ-augmented neural networks: Ensemble neural networks (ENN), Bayesian neural networks (BNN), Dropout neural networks (D-NN), and Gaussian neural networks (G-NN). We evaluate UQ accuracy (distinct from model accuracy) using two metrics: the distribution of normalized residuals on validation data, and the distribution of estimated uncertainties. We apply these metrics to two model data sets, representative of complex dynamical systems: an ocean engineering problem in which a ship traverses irregular wave episodes, and a dispersive wave turbulence system with extreme events, the Majda-McLaughlin-Tabak model. We present conclusions concerning model architecture and hyperparameter tuning.