mlsanityData to 7 Oct 2026

From SGD to Spectra: A Theory of Neural Network Weight Dynamics

arXiv 2507.12709 ↗First posted 17 Jul 2025cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have revolutionized machine learning, yet their training dynamics remain theoretically unclear-we develop a continuous-time, matrix-valued stochastic differential equation (SDE) framework that rigorously connects the microscopic dynamics of SGD to the macroscopic evolution of singular-value spectra in weight matrices. We derive exact SDEs showing that squared singular values follow Dyson Brownian motion with eigenvalue repulsion, and characterize stationary distributions as gamma-type densities with power-law tails, providing the first theoretical explanation for the empirically observed 'bulk+tail' spectral structure in trained networks. Through controlled experiments on transformer and MLP architectures, we validate our theoretical predictions and demonstrate quantitative agreement between SDE-based forecasts and observed spectral evolution, providing a rigorous foundation for understanding why deep learning works.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

15 of 29 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  10. 2017Three Factors Influencing Minima in SGDcited by 270
  11. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  12. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
3 more
  1. 2019A Mean Field Theory of Batch Normalizationcited by 58
  2. 2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
  3. 2024Stochastic weight matrix dynamics during learning and Dyson Brownian motioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  2. 2026Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkincited by 2
  3. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 2
  4. 2026Rethinking Neural Network Learning Rates: A Stackelberg Perspectivecited by 1
  5. 2026Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selectioncited by 0
  6. 2026SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updatescited by 0
  7. 2026The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetrycited by 0
  8. 2026Spectral Tail Interventions in Decoder-Only Language Models: Reasoning-Sensitive Weight Structure from Controlled Surgerycited by 0