mlsanityData to 13 Oct 2026

Bayesian Neural Network Priors Revisited

arXiv 2102.06571 ↗First posted 12 Feb 2021stat.ML cs.LGcited by 121

Not on the map. Why?

Abstract (arXiv)

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, it is unclear whether these priors accurately reflect our true beliefs about the weight distributions or give optimal performance. To find better priors, we study summary statistics of neural network weights in networks trained using stochastic gradient descent (SGD). We find that convolutional neural network (CNN) and ResNet weights display strong spatial correlations, while fully connected networks (FCNNs) display heavy-tailed weight distributions. We show that building these observations into priors can lead to improved performance on a variety of image classification datasets. Surprisingly, these priors mitigate the cold posterior effect in FCNNs, but slightly increase the cold posterior effect in ResNets.

Key facts. Cited by 121 papers in the map (data through 13 Oct 2026). Builds on 50 of its 106 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

50 of 106 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,430
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,328
  6. 2013Auto-Encoding Variational Bayescited by 4,024
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,340
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,278
  9. 2017Deep Neural Networks as Gaussian Processescited by 618
  10. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  11. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  12. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 316
38 more
  1. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  2. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  3. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  4. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 166
  6. 2019Practical Deep Learning with Bayesian Principlescited by 158
  7. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  8. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 150
  9. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  10. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  11. 2019Functional Variational Bayesian Neural Networkscited by 127
  12. 2020The Heavy-Tail Phenomenon in SGDcited by 123
  13. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  14. 2019Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMCcited by 90
  15. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  16. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  17. 2020On the role of data in PAC-Bayes boundscited by 64
  18. 2020PAC-Bayes Analysis Beyond the Usual Boundscited by 63
  19. 2020All You Need is a Good Functional Prior for Bayesian Deep Learningcited by 61
  20. 2020A statistical theory of cold posteriors in deep neural networkscited by 54
  21. 2017Model Selection in Bayesian Neural Networks via Horseshoe Priorscited by 52
  22. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  23. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  24. 2018Variational Implicit Processescited by 41
  25. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  26. 2020Deep kernel processescited by 34
  27. 2021Exact Langevin Dynamics with Stochastic Gradientscited by 30
  28. 2020Stable behaviour of infinitely wide deep neural networkscited by 23
  29. 2018Overpruning in Variational Bayesian Neural Networkscited by 22
  30. 2019Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functionscited by 22
  31. 2020The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networkscited by 20
  32. 2019Bayesian Inference for Large Scale Image Classificationcited by 18
  33. 2020Hierarchical Gaussian Process Priors for Bayesian Neural Network Weightscited by 13
  34. 2018The Deep Weight Priorcited by 11
  35. 2019Richer priors for infinitely wide multi-layer perceptronscited by 9
  36. 2019Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Carecited by 6
  37. 2015A Scale Mixture Perspective of Multiplicative Noise in Neural Networkscited by 4
  38. 2021Correlated Weights in Infinite Limits of Deep Convolutional Neural Networkscited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 23; 2022: 26; 2023: 26; 2024: 24; 2025: 10; 2026: 12.

121 papers.

Sort:

By area:

  1. 2021Transformers Can Do Bayesian Inferencecited by 346
  2. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  3. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 114
  4. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  5. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  6. 2024Variational Bayesian Last Layerscited by 68
  7. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  8. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  9. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  10. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  11. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  12. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
38 more
  1. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  2. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  3. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  4. 2022SAM as an Optimal Relaxation of Bayescited by 41
  5. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  6. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  7. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  8. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  9. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  10. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  11. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  12. 2023Uncertainty-aware Surrogate Models for Airfoil Flow Simulations with Denoising Diffusion Probabilistic Modelscited by 22
  13. 2021On Stein Variational Neural Network Ensemblescited by 21
  14. 2021Periodic Activation Functions Induce Stationaritycited by 21
  15. 2022Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learningcited by 21
  16. 2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
  17. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  18. 2021Neural Variational Gradient Descentcited by 19
  19. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  20. 2022UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographycited by 19
  21. 2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
  22. 2022Variational Model Perturbation for Source-Free Domain Adaptationcited by 18
  23. 2021Bayesian OOD detection with aleatoric uncertainty and outlier exposurecited by 16
  24. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  25. 2021Pathologies in priors and inference for Bayesian transformerscited by 13
  26. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  27. 2022How Tempering Fixes Data Augmentation in Bayesian Neural Networkscited by 13
  28. 2022Sampling in Constrained Domains with Orthogonal-Space Variational Gradient Descentcited by 11
  29. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  30. 2022Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Priorcited by 10
  31. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  32. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  33. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 10
  34. 2025Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networkscited by 10
  35. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  36. 2023Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Conceptscited by 9
  37. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  38. 2024Implicit Neural Representations for Robust Joint Sparse-View CT Reconstructioncited by 9

50 most cited of 121 All 121 (JSON)