mlsanityData to 14 Oct 2026

Depth induces scale-averaging in overparameterized linear Bayesian neural networks

arXiv 2111.11954 ↗First posted 23 Nov 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Inference in deep Bayesian neural networks is only fully understood in the infinite-width limit, where the posterior flexibility afforded by increased depth washes out and the posterior predictive collapses to a shallow Gaussian process. Here, we interpret finite deep linear Bayesian neural networks as data-dependent scale mixtures of Gaussian process predictors across output channels. We leverage this observation to study representation learning in these networks, allowing us to connect limiting results obtained in previous studies within a unified framework. In total, these results advance our analytical understanding of how depth affects inference in a simple class of Bayesian neural networks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 23 Nov 2021.

Builds on

19 of 40 references.

References · 19

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  6. 2021The Principles of Deep Learning Theorycited by 173
  7. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  8. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  9. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  10. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  11. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  12. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
7 more
  1. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  2. 2020Deep kernel processescited by 34
  3. 2020Exact posterior distributions of wide Bayesian neural networkscited by 30
  4. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  5. 2021Statistical limits of dictionary learning: random matrix theory and the spectral replica methodcited by 10
  6. 2021Scale Mixtures of Neural Network Gaussian Processescited by 5
  7. 2016The Matrix Generalized Inverse Gaussian Distribution: Properties and Applicationscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 2.

7 papers.

  1. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  2. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  3. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  4. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  5. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  6. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 10
  7. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9