mlsanityData to 14 Oct 2026

Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theory

arXiv 2510.01930 ↗First posted 2 Oct 2025stat.ML cond-mat.dis-nn cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Diagonal linear networks (DLNs) are a tractable model that captures several nontrivial behaviors in neural network training, such as initialization-dependent solutions and incremental learning. These phenomena are typically studied in isolation, leaving the overall dynamics insufficiently understood. In this work, we present a unified analysis of various phenomena in the gradient flow dynamics of DLNs. Using Dynamical Mean-Field Theory (DMFT), we derive a low-dimensional effective process that captures the asymptotic gradient flow dynamics in high dimensions. Analyzing this effective process yields new insights into DLN dynamics, including loss convergence rates and their trade-off with generalization, and systematically reproduces many of the previously observed phenomena. These findings deepen our understanding of DLNs and demonstrate the effectiveness of the DMFT approach in analyzing high-dimensional learning dynamics of neural networks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

36 of 59 references.

References · 36

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
24 more
  1. 2021Deep learning: a statistical viewpointcited by 240
  2. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  3. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  4. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  5. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  6. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  7. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  8. 2017Implicit Regularization in Deep Learningcited by 90
  9. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  10. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  11. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  12. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
  13. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  14. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  15. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  16. 2021A unifying tutorial on Approximate Message Passingcited by 50
  17. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  18. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  19. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  20. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  21. 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 25
  22. 2018Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inferencecited by 24
  23. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 19
  24. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descentcited by 0
  2. 2026Statistical Vignettes on the Grokking Phenomenoncited by 0