Proportional infinite-width infinite-depth limit for deep linear neural networks
Not on the map. Why?
Abstract (arXiv)
We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 22 Nov 2024.
Builds on
21 of 38 references.
References · 21
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
- 2018A mathematical theory of semantic development in deep neural networkscited by 186
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
- 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
9 more
- 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
- 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
- 2022Bayesian Interpolation with Deep Linear Networkscited by 21
- 2022Infinite-width limit of deep linear neural networkscited by 20
- 2023Quantitative CLTs in Deep Neural Networkscited by 20
- 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
- 2023Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processescited by 13
- 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
- 2023Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functionscited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.