mlsanityData to 7 Oct 2026

Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks

arXiv 2002.10118 ↗First posted 24 Feb 2020stat.ML cs.LGcited by 227

Not on the map. Why?

Abstract (arXiv)

The point estimates of ReLU classification networks---arguably the most widely used neural network architecture---have been shown to yield arbitrarily high confidence far away from the training data. This architecture, in conjunction with a maximum a posteriori estimation scheme, is thus not calibrated nor robust. Approximate Bayesian inference has been empirically demonstrated to improve predictive uncertainty in neural networks, although the theoretical analysis of such Bayesian approximations is limited. We theoretically analyze approximate Gaussian distributions on the weights of ReLU networks and show that they fix the overconfidence problem. Furthermore, we show that even a simplistic, thus cheap, Bayesian approximation, also fixes these issues. This indicates that a sufficient condition for a calibrated uncertainty on a ReLU network is "to be a bit Bayesian". These theoretical results validate the usage of last-layer Bayesian approximation and motivate a range of a fidelity-cost trade-off. We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.

Key facts. Cited by 227 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 32; 2023: 44; 2024: 46; 2025: 35; 2026: 31.

227 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  4. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  5. 2023Non-Parametric Outlier Synthesiscited by 129
  6. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  7. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  8. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  9. 2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 79
  10. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  11. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  12. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 71
38 more
  1. 2024Variational Bayesian Last Layerscited by 67
  2. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  3. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  4. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  5. 2021DOCTOR: A Simple Method for Detecting Misclassification Errorscited by 52
  6. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  7. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  8. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 47
  9. 2022Uncertainty-guided Source-free Domain Adaptationcited by 46
  10. 2021LiBRe: A Practical Bayesian Approach to Adversarial Detectioncited by 40
  11. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
  12. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  13. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  14. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  15. 2023BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inferencecited by 30
  16. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  17. 2021A Bit More Bayesian: Domain-Invariant Learning with Uncertaintycited by 27
  18. 2021Deep Ensembles from a Bayesian Perspectivecited by 27
  19. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  20. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  21. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 27
  22. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  23. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  24. 2022Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networkscited by 24
  25. 2022Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEscited by 24
  26. 2021Bayesian Deep Basis Fitting for Depth Completion with Uncertaintycited by 23
  27. 2021Periodic Activation Functions Induce Stationaritycited by 21
  28. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  29. 2021Label-Assemble: Leveraging Multiple Datasets with Partial Labelscited by 20
  30. 2025Generative Uncertainty in Diffusion Modelscited by 20
  31. 2021On Margins and Derandomisation in PAC-Bayescited by 19
  32. 2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
  33. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  34. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  35. 2021Provably Robust Detection of Out-of-distribution Data (almost) for freecited by 17
  36. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  37. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  38. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15

50 most cited of 227 All 227 (JSON)