mlsanityData to 7 Oct 2026

Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling

arXiv 2206.07673 ↗First posted 15 Jun 2022stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce repriorisation, a data-dependent reparameterisation which transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow. The repriorisation map acts directly on parameters, and its analytic simplicity complements the known neural network Gaussian process (NNGP) behaviour of wide BNNs in function space. Exploiting the repriorisation, we develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN. This contrasts with the typically poor performance of MCMC in high dimensions. We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks. Improvements are achieved at all widths, with the margin between reparametrised and standard BNNs growing with layer width.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

38 of 64 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2016Wide Residual Networkscited by 3,057
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2017Deep Neural Networks as Gaussian Processescited by 617
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
26 more
  1. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  2. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  3. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  4. 2021The Principles of Deep Learning Theorycited by 169
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  6. 2019Practical Deep Learning with Bayesian Principlescited by 158
  7. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  8. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  9. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  10. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  11. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  12. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  13. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  14. 2019NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transportcited by 73
  15. 2015On Sparse variational methods and the Kullback-Leibler divergence between stochastic processescited by 56
  16. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  17. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  18. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  19. 2020Deep kernel processescited by 34
  20. 2020Exact posterior distributions of wide Bayesian neural networkscited by 29
  21. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  22. 2020Wide Neural Networks with Bottlenecks are Deep Gaussian Processescited by 18
  23. 2016An introduction to sampling via measure transportcited by 16
  24. 2019Information in Infinite Ensembles of Infinitely-Wide Neural Networkscited by 14
  25. 2017Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributionscited by 7
  26. 2021Computing the Information Content of Trained Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 2; 2026: 1.

9 papers.

  1. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  2. 2023Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Modelscited by 15
  3. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  4. 2024Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activationscited by 2
  5. 2024Function-Space MCMC for Bayesian Wide Neural Networkscited by 1
  6. 2023Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networkscited by 0
  7. 2025Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networkscited by 0
  8. 2025Hi-fi functional priors by learning activationscited by 0
  9. 2026Kinetic Langevin Splitting Schemes for Constrained Samplingcited by 0