mlsanityData to 7 Oct 2026

Bayesian Deep Learning and a Probabilistic Perspective of Generalization

arXiv 2002.08791 ↗First posted 20 Feb 2020cs.LG stat.MLcited by 480

Not on the map. Why?

Abstract (arXiv)

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. We also show that Bayesian model averaging alleviates double descent, resulting in monotonic performance improvements with increased flexibility. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.

Key facts. Cited by 480 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 79; 2022: 95; 2023: 102; 2024: 87; 2025: 64; 2026: 53.

480 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  4. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  5. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  6. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representationscited by 179
  7. 2023Protein Design with Guided Discrete Diffusioncited by 154
  8. 2021Bayesian Neural Network Priors Revisitedcited by 121
  9. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  10. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  11. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  12. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
38 more
  1. 2022SELFIES and the future of molecular string representationscited by 76
  2. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  3. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  4. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  5. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  6. 2024Variational Bayesian Last Layerscited by 67
  7. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  8. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  9. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  10. 2023Uncertainty Estimation by Fisher Information-based Evidential Deep Learningcited by 58
  11. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  12. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  13. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  14. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  15. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  16. 2021Quantifying Uncertainty in Deep Spatiotemporal Forecastingcited by 48
  17. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 47
  18. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  19. 2021Posterior Meta-Replay for Continual Learningcited by 45
  20. 2023Sampling-Based Accuracy Testing of Posterior Estimators for General Inferencecited by 45
  21. 2022Investigating the Impact of Model Misspecification in Neural Simulation-based Inferencecited by 44
  22. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  23. 2023PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awarenesscited by 42
  24. 2021LiBRe: A Practical Bayesian Approach to Adversarial Detectioncited by 40
  25. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  26. 2021Correlated Input-Dependent Label Noise in Large-Scale Image Classificationcited by 37
  27. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  28. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  29. 2022Discovering and forecasting extreme events via active learning in neural operatorscited by 32
  30. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  31. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  32. 2021AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantificationcited by 31
  33. 2023A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methodscited by 31
  34. 2024Improving Uncertainty Quantification in Large Language Models via Semantic Embeddingscited by 30
  35. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  36. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  37. 2023Introducing an Improved Information-Theoretic Measure of Predictive Uncertaintycited by 28
  38. 2021Deep Ensembles from a Bayesian Perspectivecited by 27

50 most cited of 480 All 480 (JSON)