mlsanityData to 7 Oct 2026

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer

arXiv 2502.02531 ↗First posted 4 Feb 2025cs.LG cond-mat.dis-nn stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

We theoretically characterize gradient descent dynamics in deep linear networks trained at large width from random initialization and on large quantities of random data. Our theory captures the ``wider is better" effect of mean-field/maximum-update parameterized networks as well as hyperparameter transfer effects, which can be contrasted with the neural-tangent parameterization where optimal learning rates shift with model width. We provide asymptotic descriptions of both non-residual and residual neural networks, the latter of which enables an infinite depth limit when branches are scaled as 1/√depth. We also compare training with one-pass stochastic gradient descent to the dynamics when training data are repeated at each iteration. Lastly, we show that this model recovers the accelerated power law training dynamics for power law structured data in the rich regime observed in recent works.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

39 of 53 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  9. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  10. 2021The Principles of Deep Learning Theorycited by 169
  11. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  12. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
27 more
  1. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  3. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  4. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  5. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  6. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
  7. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  8. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  9. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  10. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 71
  11. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  12. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  13. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  14. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  15. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  16. 2022Dimension free ridge regressioncited by 50
  17. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  18. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  19. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  20. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  21. 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
  22. 2021The effective noise of Stochastic Gradient Descentcited by 25
  23. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  24. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  25. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  26. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  27. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 10.

17 papers.

Sort:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  2. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  3. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  4. 2025A Proof of Learning Rate Transfer under μPcited by 3
  5. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 2
  6. 2025Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticitycited by 2
  7. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  8. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  9. 2025On residual network depthcited by 1
  10. 2026Depth, Not Data: An Analysis of Hessian Spectral Bifurcationcited by 1
  11. 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networkscited by 1
  12. 2026A unified theory of feature learning in RNNs and DNNscited by 1
5 more
  1. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  2. 2026Hidden Gauge Controls Feature Specialization in ReLU Networkscited by 1
  3. 2026Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networkscited by 0
  4. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
  5. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0