# NOMU: Neural Optimization-based Model Uncertainty

arXiv: https://arxiv.org/abs/2102.13640 · HTML: https://mlsanity.com/paper/2102.13640/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 27 of 27)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)
- [Bayesian Optimization of Combinatorial Structures](https://mlsanity.com/paper/1806.08838/) (2018)
- [Function Space Particle Optimization for Bayesian Neural Networks](https://mlsanity.com/paper/1902.09754/) (2019)
- [Benchmarking the Neural Linear Model for Regression](https://mlsanity.com/paper/1912.08416/) (2019)
- [Regression Prior Networks](https://mlsanity.com/paper/2006.11590/) (2020)
- [On the Importance of Strong Baselines in Bayesian Deep Learning](https://mlsanity.com/paper/1811.09385/) (2018)
- [Deep Learning-powered Iterative Combinatorial Auctions](https://mlsanity.com/paper/1907.05771/) (2019)

## Built on by (most cited first, 16 of 16)

- [Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents](https://mlsanity.com/paper/2505.22655/) (2025)
- [Uncertainty Quantification Metrics for Deep Regression](https://mlsanity.com/paper/2405.04278/) (2024)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Bayesian Optimization-based Combinatorial Assignment](https://mlsanity.com/paper/2208.14698/) (2022)
- [CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk](https://mlsanity.com/paper/2507.08150/) (2025)
- [Universal Regular Conditional Distributions](https://mlsanity.com/paper/2105.07743/) (2021)
- [How Infinitely Wide Neural Networks Can Benefit from Multi-task Learning -- an Exact Macroscopic Characterization](https://mlsanity.com/paper/2112.15577/) (2021)
- [Machine Learning-Powered Combinatorial Clock Auction](https://mlsanity.com/paper/2308.10226/) (2023)
- [Machine Learning-Powered Course Allocation](https://mlsanity.com/paper/2210.00954/) (2022)
- [How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer](https://mlsanity.com/paper/2303.11454/) (2023)
- [CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Models](https://mlsanity.com/paper/2306.03598/) (2023)
- [Learning Generative Models for Climbing Aircraft from Radar Data](https://mlsanity.com/paper/2309.14941/) (2023)
- [Toward Robust Uncertainty Estimation with Random Activation Functions](https://mlsanity.com/paper/2302.14552/) (2023)
- [Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing](https://mlsanity.com/paper/2306.06599/) (2023)
- [Deterministic Langevin Unconstrained Optimization with Normalizing Flows](https://mlsanity.com/paper/2310.00745/) (2023)
- [Scalable neural network-based blackbox optimization](https://mlsanity.com/paper/2508.03827/) (2025)

## Abstract (arXiv)

We study methods for estimating model uncertainty for neural networks (NNs) in regression. To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce a new approach for capturing model uncertainty for NNs, which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea of NOMU is to design a network architecture consisting of two connected sub-NNs, one for model prediction and one for model uncertainty, and to train it using a carefully-designed loss function. Importantly, our design enforces that NOMU satisfies our five desiderata. Due to its modular architecture, NOMU can provide model uncertainty for any given (previously trained) NN if given access to its training data. We evaluate NOMU in various regressions tasks and noiseless Bayesian optimization (BO) with costly evaluations. In regression, NOMU performs at least as well as state-of-the-art methods. In BO, NOMU even outperforms all considered benchmarks.