# Wide Mean-Field Bayesian Neural Networks Ignore the Data

arXiv: https://arxiv.org/abs/2202.11670 · HTML: https://mlsanity.com/paper/2202.11670/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Bayesian Deep Ensembles via the Neural Tangent Kernel](https://mlsanity.com/paper/2007.05864/) (2020)
- [Exact posterior distributions of wide Bayesian neural networks](https://mlsanity.com/paper/2006.10541/) (2020)
- [Overpruning in Variational Bayesian Neural Networks](https://mlsanity.com/paper/1801.06230/) (2018)

## Built on by (most cited first, 22 of 22)

- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [SAM as an Optimal Relaxation of Bayes](https://mlsanity.com/paper/2210.01620/) (2022)
- [Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients](https://mlsanity.com/paper/2206.06295/) (2022)
- [On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective](https://mlsanity.com/paper/2310.00137/) (2023)
- [Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2402.01476/) (2024)
- [Streamlining Prediction in Bayesian Deep Learning](https://mlsanity.com/paper/2411.18425/) (2024)
- [Variational Visual Question Answering for Uncertainty-Aware Selective Prediction](https://mlsanity.com/paper/2505.09591/) (2025)
- [Challenges and Pitfalls of Bayesian Unlearning](https://mlsanity.com/paper/2207.03227/) (2022)
- [Identifying Causal Direction via Variational Bayesian Compression](https://mlsanity.com/paper/2505.07503/) (2025)
- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [An Empirical Analysis of the Advantages of Finite- v.s. Infinite-Width Bayesian Neural Networks](https://mlsanity.com/paper/2211.09184/) (2022)
- [Variational Bayesian Neural Networks via Resolution of Singularities](https://mlsanity.com/paper/2302.06035/) (2023)
- [Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference](https://mlsanity.com/paper/2307.04779/) (2023)
- [Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networks](https://mlsanity.com/paper/2503.11808/) (2025)
- [Structured Partial Stochasticity in Bayesian Neural Networks](https://mlsanity.com/paper/2405.17666/) (2024)
- [Central Limit Theorem for Bayesian Neural Network trained with Variational Inference](https://mlsanity.com/paper/2406.09048/) (2024)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [Variational Deep Learning via Implicit Regularization](https://mlsanity.com/paper/2505.20235/) (2025)
- [Occam's Razor is Only as Sharp as Your ELBO](https://mlsanity.com/paper/2604.25984/) (2026)
- [Gaussian Mean Field Variational Inference can Overestimate Predictive Variance](https://mlsanity.com/paper/2606.25745/) (2026)
- [Efficient Bayesian Deep Ensembles via Analytic Predictive Inference](https://mlsanity.com/paper/2607.06776/) (2026)

## Abstract (arXiv)

Bayesian neural networks (BNNs) combine the expressive power of deep learning with the advantages of Bayesian formalism. In recent years, the analysis of wide, deep BNNs has provided theoretical insight into their priors and posteriors. However, we have no analogous insight into their posteriors under approximate inference. In this work, we show that mean-field variational inference entirely fails to model the data when the network width is large and the activation function is odd. Specifically, for fully-connected BNNs with odd activation functions and a homoscedastic Gaussian likelihood, we show that the optimal mean-field variational posterior predictive (i.e., function space) distribution converges to the prior predictive distribution as the width tends to infinity. We generalize aspects of this result to other likelihoods. Our theoretical results are suggestive of underfitting behavior previously observered in BNNs. While our convergence bounds are non-asymptotic and constants in our analysis can be computed, they are currently too loose to be applicable in standard training regimes. Finally, we show that the optimal approximate posterior need not tend to the prior if the activation function is not odd, showing that our statements cannot be generalized arbitrarily.