mlsanityData to 14 Oct 2026

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

arXiv 2605.07870 ↗First posted 8 May 2026cond-mat.dis-nn cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles whose spike directions remain statistically dependent on the random bulk. We apply this framework to two settings: (1) infinite-width nonlinear networks in mean-field/μP scaling and (2) deep linear networks in the proportional high-dimensional limit, where width, input dimension, and sample size diverge with fixed ratios. Our theory predicts how outliers evolve with training time, width, output scale, and initialization variance. In deep linear networks, μP yields width-consistent outlier dynamics and hyperparameter transfer, including width-stable growth of the leading NTK mode toward the edge of stability (EoS). In contrast, NTK parameterization exhibits strongly width-dependent outlier dynamics, despite converging to a stable large-width limit. We show that this bulk+outlier picture is descriptive of simple tasks with small output channels, but that tasks involving large numbers of outputs (ImageNet classification or GPT language modeling) are better described by a restructuring of the spectral bulk. We develop a toy model with extensive output channels that recapitulates this phenomenon and show that edge of the spectrum still converges for sufficiently wide networks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 76 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

33 of 76 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2021Explaining Neural Scaling Lawscited by 363
  7. 20242 OLMo 2 Furiouscited by 340
  8. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  9. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  10. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  11. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  12. 2018A Walk with SGDcited by 78
21 more
  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  2. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  3. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  4. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  5. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  6. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  7. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  8. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  9. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  10. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 35
  11. 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 30
  12. 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 25
  13. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  14. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 19
  15. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  16. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  17. 2026ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limitcited by 10
  18. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 7
  19. 2026High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networkscited by 7
  20. 2026Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learningcited by 5
  21. 2022Rigorous dynamical mean field theory for stochastic gradient descent methodscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026A Defense of the Quadratic Modelcited by 14
  2. 2026The Spectral Dynamics and Noise Geometry of Muoncited by 3
  3. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 2
  4. 2026A Function-Space Approach to the Statistical Mechanics of Learning Dynamicscited by 0
  5. 2026Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizonscited by 0
  6. 2026A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scalingcited by 0