mlsanityData to 7 Oct 2026

Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers

arXiv 2406.03260 ↗First posted 5 Jun 2024stat.ML cond-mat.dis-nn cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Deep linear networks have been extensively studied, as they provide simplified models of deep learning. However, little is known in the case of finite-width architectures with multiple outputs and convolutional layers. In this manuscript, we provide rigorous results for the statistics of functions implemented by the aforementioned class of networks, thus moving closer to a complete characterization of feature learning in the Bayesian setting. Our results include: (i) an exact and elementary non-asymptotic integral representation for the joint prior distribution over the outputs, given in terms of a mixture of Gaussians; (ii) an analytical formula for the posterior distribution in the case of squared error loss function (Gaussian likelihood); (iii) a quantitative description of the feature learning infinite-width regime, using large deviation theory. From a physical perspective, deep architectures with multiple outputs or convolutional layers represent different manifestations of kernel shape renormalization, and our work provides a dictionary that translates this physics intuition and terminology into rigorous Bayesian statistics.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 41 of its 69 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

41 of 69 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  10. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  11. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  12. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
29 more
  1. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  2. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  3. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  4. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  5. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  6. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  7. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  8. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  9. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  10. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  11. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  12. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  13. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  14. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  15. 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
  16. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  17. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  18. 2022Infinite-width limit of deep linear neural networkscited by 20
  19. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  20. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  21. 2022Globally Gated Deep Linear Networkscited by 12
  22. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  23. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  24. 2023Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processescited by 11
  25. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  26. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  27. 2024Wilsonian Renormalization of Neural Network Gaussian Processescited by 8
  28. 2023Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisationcited by 6
  29. 2024Random features and polynomial rulescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 2.

12 papers.

  1. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
  2. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  3. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  4. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  5. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  6. 2024Proportional infinite-width infinite-depth limit for deep linear neural networkscited by 3
  7. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 2
  8. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  9. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  10. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  11. 2025Deep Neural Nets as Hamiltonianscited by 0
  12. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0