# A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness

arXiv: https://arxiv.org/abs/2205.00403 · HTML: https://mlsanity.com/paper/2205.00403/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 97 of its 156 references that are in the map.
- First posted on arXiv 1 May 2022.

## Builds on (most cited first, 30 of 97)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)

## Built on by (most cited first, 30 of 49)

- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Assaying Out-Of-Distribution Generalization in Transfer Learning](https://mlsanity.com/paper/2207.09239/) (2022)
- [Improving Zero-shot Generalization and Robustness of Multi-modal Models](https://mlsanity.com/paper/2212.01758/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Variational Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2302.12565/) (2023)
- [Uncertainty Quantification Metrics for Deep Regression](https://mlsanity.com/paper/2405.04278/) (2024)
- [Open-Vocabulary Calibration for Fine-tuned CLIP](https://mlsanity.com/paper/2402.04655/) (2024)
- [Ask a Strong LLM Judge when Your Reward Model is Uncertain](https://mlsanity.com/paper/2510.20369/) (2025)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles](https://mlsanity.com/paper/2412.15758/) (2024)
- [Raising the Bar on the Evaluation of Out-of-Distribution Detection](https://mlsanity.com/paper/2209.11960/) (2022)
- [Beyond AUROC & co. for evaluating out-of-distribution detection performance](https://mlsanity.com/paper/2306.14658/) (2023)
- [Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification](https://mlsanity.com/paper/2304.04042/) (2023)
- [Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification](https://mlsanity.com/paper/2404.08981/) (2024)
- [Morse Neural Networks for Uncertainty Quantification](https://mlsanity.com/paper/2307.00667/) (2023)
- [Influence of uncertainty estimation techniques on false-positive reduction in liver lesion detection](https://mlsanity.com/paper/2206.10911/) (2022)
- [Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks](https://mlsanity.com/paper/2305.17063/) (2023)
- [A Rate-Distortion View of Uncertainty Quantification](https://mlsanity.com/paper/2406.10775/) (2024)
- [Efficient Post-Hoc Uncertainty Calibration via Variance-Based Smoothing](https://mlsanity.com/paper/2503.15583/) (2025)
- [Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations](https://mlsanity.com/paper/2601.08064/) (2026)
- [Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations](https://mlsanity.com/paper/2210.06112/) (2022)
- [Pushing the Accuracy-Group Robustness Frontier with Introspective Self-play](https://mlsanity.com/paper/2302.05807/) (2023)
- [ScatterUQ: Interactive Uncertainty Visualizations for Multiclass Deep Learning Problems](https://mlsanity.com/paper/2308.04588/) (2023)
- [Transferable Candidate Proposal with Bounded Uncertainty](https://mlsanity.com/paper/2312.04604/) (2023)
- [Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning](https://mlsanity.com/paper/2412.04177/) (2024)
- [Learning Hyperparameters via a Data-Emphasized Variational Objective](https://mlsanity.com/paper/2502.01861/) (2025)
- [Old Rules in a New Game: Mapping Uncertainty Quantification to Quantum Machine Learning](https://mlsanity.com/paper/2507.14919/) (2025)

## Abstract (arXiv)

Accurate uncertainty quantification is a major challenge in deep learning, as neural networks can make overconfident errors and assign high confidence predictions to out-of-distribution (OOD) inputs. The most popular approaches to estimate predictive uncertainty in deep learning are methods that combine predictions from multiple neural networks, such as Bayesian neural networks (BNNs) and deep ensembles. However their practicality in real-time, industrial-scale applications are limited due to the high memory and computational cost. Furthermore, ensembles and BNNs do not necessarily fix all the issues with the underlying member networks. In this work, we study principled approaches to improve uncertainty property of a single network, based on a single, deterministic representation. By formalizing the uncertainty quantification as a minimax learning problem, we first identify distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs with two simple changes: (1) applying spectral normalization to hidden weights to enforce bi-Lipschitz smoothness in representations and (2) replacing the last output layer with a Gaussian process layer. On a suite of vision and language understanding benchmarks, SNGP outperforms other single-model approaches in prediction, calibration and out-of-domain detection. Furthermore, SNGP provides complementary benefits to popular techniques such as deep ensembles and data augmentation, making it a simple and scalable building block for probabilistic deep learning. Code is open-sourced at https://github.com/google/uncertainty-baselines