# Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers

arXiv: https://arxiv.org/abs/2406.03260 · HTML: https://mlsanity.com/paper/2406.03260/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 69 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 30 of 41)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Generalisation error in learning with random features and the hidden manifold model](https://mlsanity.com/paper/2002.09339/) (2020)
- [The Gaussian equivalence of generative models for learning with shallow neural networks](https://mlsanity.com/paper/2006.14709/) (2020)
- [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018)
- [Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks](https://mlsanity.com/paper/2205.09653/) (2022)
- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Random Neural Networks in the Infinite Width Limit as Gaussian Processes](https://mlsanity.com/paper/2107.01562/) (2021)
- [The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization](https://mlsanity.com/paper/2206.02768/) (2022)
- [Bayes-optimal Learning of Deep Random Networks of Extensive-width](https://mlsanity.com/paper/2302.00375/) (2023)
- [A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs](https://mlsanity.com/paper/2106.04110/) (2021)
- [Data-driven emergence of convolutional structure in neural networks](https://mlsanity.com/paper/2202.00565/) (2022)
- [Gaussian Universality of Perceptrons with Random Labels](https://mlsanity.com/paper/2205.13303/) (2022)
- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies](https://mlsanity.com/paper/2204.01058/) (2022)
- [Learning sparse features can lead to overfitting in neural networks](https://mlsanity.com/paper/2206.12314/) (2022)
- [Bayesian Interpolation with Deep Linear Networks](https://mlsanity.com/paper/2212.14457/) (2022)
- [Infinite-width limit of deep linear neural networks](https://mlsanity.com/paper/2211.16980/) (2022)

## Built on by (most cited first, 12 of 12)

- [Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer](https://mlsanity.com/paper/2502.02531/) (2025)
- [Adaptive kernel predictors from feature-learning infinite limits of neural networks](https://mlsanity.com/paper/2502.07998/) (2025)
- [From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning](https://mlsanity.com/paper/2502.03210/) (2025)
- [Statistical mechanics of extensive-width Bayesian neural networks near interpolation](https://mlsanity.com/paper/2505.24849/) (2025)
- [Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime](https://mlsanity.com/paper/2505.03577/) (2025)
- [Proportional infinite-width infinite-depth limit for deep linear neural networks](https://mlsanity.com/paper/2411.15267/) (2024)
- [Transfer Learning in Infinite Width Feature Learning Networks](https://mlsanity.com/paper/2507.04448/) (2025)
- [Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation](https://mlsanity.com/paper/2501.18530/) (2025)
- [Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime](https://mlsanity.com/paper/2605.29684/) (2026)
- [Bayesian Inference with Shaped Deep Non-linear MLPs](https://mlsanity.com/paper/2605.30860/) (2026)
- [Deep Neural Nets as Hamiltonians](https://mlsanity.com/paper/2503.23982/) (2025)
- [Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation](https://mlsanity.com/paper/2510.24616/) (2025)

## Abstract (arXiv)

Deep linear networks have been extensively studied, as they provide simplified models of deep learning. However, little is known in the case of finite-width architectures with multiple outputs and convolutional layers. In this manuscript, we provide rigorous results for the statistics of functions implemented by the aforementioned class of networks, thus moving closer to a complete characterization of feature learning in the Bayesian setting. Our results include: (i) an exact and elementary non-asymptotic integral representation for the joint prior distribution over the outputs, given in terms of a mixture of Gaussians; (ii) an analytical formula for the posterior distribution in the case of squared error loss function (Gaussian likelihood); (iii) a quantitative description of the feature learning infinite-width regime, using large deviation theory. From a physical perspective, deep architectures with multiple outputs or convolutional layers represent different manifestations of kernel shape renormalization, and our work provides a dictionary that translates this physics intuition and terminology into rigorous Bayesian statistics.