mlsanityData to 7 Oct 2026

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

arXiv 2006.10108 ↗First posted 17 Jun 2020cs.LG stat.MLcited by 389

Not on the map. Why?

Abstract (arXiv)

Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost. This motivates us to study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN). By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer with a Gaussian process. On a suite of vision and language understanding tasks and on modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep ensembles in prediction, calibration and out-of-domain detection, and outperforms the other single-model approaches.

Key facts. Cited by 389 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 60; 2023: 83; 2024: 77; 2025: 66; 2026: 49.

389 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  3. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 206
  4. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  6. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  7. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  8. 2022SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptationcited by 146
  9. 2023Non-Parametric Outlier Synthesiscited by 129
  10. 2021Epistemic Neural Networkscited by 120
  11. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  12. 2021Repulsive Deep Ensembles are Bayesiancited by 103
38 more
  1. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  2. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  3. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  4. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  5. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  6. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  7. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  8. 2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 79
  9. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  10. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  11. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  12. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67
  13. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  14. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  15. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  16. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  17. 2021Understanding Softmax Confidence and Uncertaintycited by 62
  18. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  19. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  20. 2023A Review of Uncertainty Estimation and its Application in Medical Imagingcited by 59
  21. 2023Towards Stability of Autoregressive Neural Operatorscited by 54
  22. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  23. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  24. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  25. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  26. 2021Posterior Meta-Replay for Continual Learningcited by 45
  27. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  28. 2021Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertaintiescited by 41
  29. 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
  30. 2021RoMA: Robust Model Adaptation for Offline Model-based Optimizationcited by 39
  31. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  32. 2023A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methodscited by 39
  33. 2024MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentationcited by 37
  34. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 37
  35. 2022Nonparametric Uncertainty Quantification for Single Deterministic Neural Networkcited by 36
  36. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
  37. 2022SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detectioncited by 33
  38. 2021No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasetscited by 31

50 most cited of 389 All 389 (JSON)