mlsanityData to 7 Oct 2026

Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference

arXiv 2307.04779 ↗First posted 10 Jul 2023stat.ML math.PR math.STcited by 1

Not on the map. Why?

Abstract (arXiv)

We provide a rigorous analysis of training by variational inference (VI) of Bayesian neural networks in the two-layer and infinite-width case. We consider a regression problem with a regularized evidence lower bound (ELBO) which is decomposed into the expected log-likelihood of the data and the Kullback-Leibler (KL) divergence between the a priori distribution and the variational posterior. With an appropriate weighting of the KL, we prove a law of large numbers for three different training schemes: (i) the idealized case with exact estimation of a multiple Gaussian integral from the reparametrization trick, (ii) a minibatch scheme using Monte Carlo sampling, commonly known as Bayes by Backprop, and (iii) a new and computationally cheaper algorithm which we introduce as Minimal VI. An important result is that all methods converge to the same mean-field limit. Finally, we illustrate our results numerically and discuss the need for the derivation of a central limit theorem.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 48 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

18 of 48 references.

References · 18

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  4. 2016Variational Inference: A Review for Statisticianscited by 1,466
  5. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  8. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  9. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  10. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  11. 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 60
  12. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
6 more
  1. 2020Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splittingcited by 54
  2. 2022Wide Mean-Field Bayesian Neural Networks Ignore the Datacited by 22
  3. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
  4. 2022Infinite-width limit of deep linear neural networkscited by 20
  5. 2013On properties of the Generalized Wasserstein distancecited by 11
  6. 2014Control to flocking of the kinetic Cucker-Smale modelcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Central Limit Theorem for Bayesian Neural Network trained with Variational Inferencecited by 0