mlsanityData to 14 Oct 2026

Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study

arXiv 2207.03859 ↗First posted 8 Jul 2022stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper studies the Variational Inference (VI) used for training Bayesian Neural Networks (BNN) in the overparameterized regime, i.e., when the number of neurons tends to infinity. More specifically, we consider overparameterized two-layer BNN and point out a critical issue in the mean-field VI training. This problem arises from the decomposition of the lower bound on the evidence (ELBO) into two terms: one corresponding to the likelihood function of the model and the second to the Kullback-Leibler (KL) divergence between the prior distribution and the variational posterior. In particular, we show both theoretically and empirically that there is a trade-off between these two terms in the overparameterized regime only when the KL is appropriately re-scaled with respect to the ratio between the the number of observations and neurons. We also illustrate our theoretical results with numerical experiments that highlight the critical choice of this ratio.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 8 Jul 2022.

Builds on

38 of 57 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  6. 2016Variational Inference: A Review for Statisticianscited by 1,472
  7. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  11. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
  12. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
26 more
  1. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  2. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  3. 2017A Conceptual Introduction to Hamiltonian Monte Carlocited by 209
  4. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
  5. 2019Practical Deep Learning with Bayesian Principlescited by 158
  6. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  7. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  8. 2019Functional Variational Bayesian Neural Networkscited by 127
  9. 2021Bayesian Neural Network Priors Revisitedcited by 121
  10. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  11. 2014Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Itcited by 95
  12. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  13. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  14. 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 61
  15. 2020All You Need is a Good Functional Prior for Bayesian Deep Learningcited by 61
  16. 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
  17. 2020Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splittingcited by 54
  18. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  19. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  20. 2021PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classescited by 31
  21. 2019Global convergence of neuron birth-death dynamicscited by 22
  22. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  23. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
  24. 2019Safe-Bayesian Generalized Linear Regressioncited by 9
  25. 2021Wide Mean-Field Variational Bayesian Neural Networks Ignore the Datacited by 7
  26. 2021Posterior Temperature Optimization in Variational Inference for Inverse Problemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 1.

5 papers.

  1. 2023Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metricscited by 8
  2. 2023On permutation symmetries in Bayesian neural network posteriors: a variational perspectivecited by 4
  3. 2024Nonparametric Automatic Differentiation Variational Inference with Spline Approximationcited by 3
  4. 2024Central Limit Theorem for Bayesian Neural Network trained with Variational Inferencecited by 0
  5. 2026The Impact of Likelihood Tempering on the Limiting Predictive Moments of Variational Bayesian Linear Neural Networkscited by 0