# Infinite-width limit of deep linear neural networks

arXiv: https://arxiv.org/abs/2211.16980 · HTML: https://mlsanity.com/paper/2211.16980/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 53 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 30 of 33)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning](https://mlsanity.com/paper/2012.09839/) (2020)
- [Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks](https://mlsanity.com/paper/1904.13262/) (2019)
- [Implicit Regularization in Deep Learning May Not Be Explainable by Norms](https://mlsanity.com/paper/2005.06398/) (2020)
- [Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity](https://mlsanity.com/paper/2106.09524/) (2021)
- [Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks](https://mlsanity.com/paper/2205.09653/) (2022)
- [Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't](https://mlsanity.com/paper/2009.10713/) (2020)
- [Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity](https://mlsanity.com/paper/2106.15933/) (2021)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [SGD Learns the Conjugate Kernel Class of the Network](https://mlsanity.com/paper/1702.08503/) (2017)
- [Stochastic Particle Gradient Descent for Infinite Ensembles](https://mlsanity.com/paper/1712.05438/) (2017)
- [Width Provably Matters in Optimization for Deep Linear Neural Networks](https://mlsanity.com/paper/1901.08572/) (2019)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime](https://mlsanity.com/paper/2005.13530/) (2020)
- [Training Linear Neural Networks: Non-Local Convergence and Complexity Results](https://mlsanity.com/paper/2002.09852/) (2020)
- [Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization](https://mlsanity.com/paper/2110.08084/) (2021)
- [Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit](https://mlsanity.com/paper/2210.12497/) (2022)

## Built on by (most cited first, 20 of 20)

- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes](https://mlsanity.com/paper/2405.17580/) (2024)
- [The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram](https://mlsanity.com/paper/2509.10167/) (2025)
- [Learning Associative Memories with Gradient Descent](https://mlsanity.com/paper/2402.18724/) (2024)
- [Universal approximation property of Banach space-valued random feature models including random neural networks](https://mlsanity.com/paper/2312.08410/) (2023)
- [Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers](https://mlsanity.com/paper/2406.03260/) (2024)
- [Learning of Population Dynamics: Inverse Optimization Meets JKO Scheme](https://mlsanity.com/paper/2506.01502/) (2025)
- [An entropy formula for the Deep Linear Network](https://mlsanity.com/paper/2509.09088/) (2025)
- [Robust Layerwise Scaling Rules by Proper Weight Decay Tuning](https://mlsanity.com/paper/2510.15262/) (2025)
- [Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size](https://mlsanity.com/paper/2506.15025/) (2025)
- [Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space](https://mlsanity.com/paper/2307.01177/) (2023)
- [Proportional infinite-width infinite-depth limit for deep linear neural networks](https://mlsanity.com/paper/2411.15267/) (2024)
- [A Proof of Learning Rate Transfer under μP](https://mlsanity.com/paper/2511.01734/) (2025)
- [Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference](https://mlsanity.com/paper/2307.04779/) (2023)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [Understanding Transfer Learning via Mean-field Analysis](https://mlsanity.com/paper/2410.17128/) (2024)
- [Towards a Principled Muon under μmathsfP: Ensuring Spectral Conditions throughout Training](https://mlsanity.com/paper/2601.01306/) (2026)
- [Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks](https://mlsanity.com/paper/2602.22925/) (2026)
- [Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit](https://mlsanity.com/paper/2605.08352/) (2026)
- [Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness](https://mlsanity.com/paper/2608.11479/) (2026)

## Abstract (arXiv)

This paper studies the infinite-width limit of deep linear neural networks initialized with random parameters. We obtain that, when the number of neurons diverges, the training dynamics converge (in a precise sense) to the dynamics obtained from a gradient descent on an infinitely wide deterministic linear neural network. Moreover, even if the weights remain random, we get their precise law along the training dynamics, and prove a quantitative convergence result of the linear predictor in terms of the number of neurons. We finally study the continuous-time limit obtained for infinitely wide linear neural networks and show that the linear predictors of the neural network converge at an exponential rate to the minimal ℓ2-norm minimizer of the risk.