mlsanityData to 7 Oct 2026

Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data

arXiv 2106.07052 ↗First posted 13 Jun 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Variational inference enables approximate posterior inference of the highly over-parameterized neural networks that are popular in modern machine learning. Unfortunately, such posteriors are known to exhibit various pathological behaviors. We prove that as the number of hidden units in a single-layer Bayesian neural network tends to infinity, the function-space posterior mean under mean-field variational inference actually converges to zero, completely ignoring the data. This is in contrast to the true posterior, which converges to a Gaussian process. Our work provides insight into the over-regularization of the KL divergence in variational inference.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 10 of its 14 references that are in the map. First posted on arXiv 13 Jun 2021.

Builds on

10 of 14 references.

References · 10

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  7. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  8. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  9. 2020Exact posterior distributions of wide Bayesian neural networkscited by 29
  10. 2018Overpruning in Variational Bayesian Neural Networkscited by 21

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 1.

7 papers.

  1. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  2. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  3. 2021Function-space Inference with Sparse Implicit Processescited by 13
  4. 2022On the detrimental effect of invariances in the likelihood for variational inferencecited by 8
  5. 2022Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical studycited by 4
  6. 2022Correcting Model Bias with Sparse Implicit Processescited by 1
  7. 2022An Overview of Uncertainty Quantification Methods for Infinite Neural Networkscited by 0