mlsanityData to 7 Oct 2026

Infinite-width limit of deep linear neural networks

arXiv 2211.16980 ↗First posted 29 Nov 2022cs.LG math.OC stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

This paper studies the infinite-width limit of deep linear neural networks initialized with random parameters. We obtain that, when the number of neurons diverges, the training dynamics converge (in a precise sense) to the dynamics obtained from a gradient descent on an infinitely wide deterministic linear neural network. Moreover, even if the weights remain random, we get their precise law along the training dynamics, and prove a quantitative convergence result of the linear predictor in terms of the number of neurons. We finally study the continuous-time limit obtained for infinitely wide linear neural networks and show that the linear predictors of the neural network converge at an exponential rate to the minimal ℓ2-norm minimizer of the risk.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

33 of 53 references.

References · 33

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  11. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  12. 2018A mathematical theory of semantic development in deep neural networkscited by 182
21 more
  1. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  2. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  4. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  5. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  6. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  7. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  8. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  9. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  10. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  11. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  12. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  13. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  14. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  15. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  16. 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
  17. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
  18. 2022Deep Linear Networks for Matrix Completion -- An Infinite Depth Limitcited by 8
  19. 2020Towards a General Theory of Infinite-Width Limits of Neural Classifierscited by 7
  20. 2021Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limitcited by 7
  21. 2021Gradient flows on graphons: existence, convergence, continuity equationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 6; 2026: 4.

20 papers.

Sort:

  1. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  2. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  3. 2024Learning Associative Memories with Gradient Descentcited by 14
  4. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  5. 2023Universal approximation property of Banach space-valued random feature models including random neural networkscited by 12
  6. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  7. 2025An entropy formula for the Deep Linear Networkcited by 8
  8. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  9. 2025Learning of Population Dynamics: Inverse Optimization Meets JKO Schemecited by 6
  10. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  11. 2025Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecited by 4
  12. 2024Proportional infinite-width infinite-depth limit for deep linear neural networkscited by 3
8 more
  1. 2025A Proof of Learning Rate Transfer under μPcited by 3
  2. 2023Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inferencecited by 1
  3. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  4. 2024Understanding Transfer Learning via Mean-field Analysiscited by 1
  5. 2026Towards a Principled Muon under μmathsfP: Ensuring Spectral Conditions throughout Trainingcited by 0
  6. 2026Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networkscited by 0
  7. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
  8. 2026Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothnesscited by 0