mlsanityData to 7 Oct 2026

Spectral phase transitions and trainability in neural network learning dynamics

arXiv 2606.28486 ↗First posted 26 Jun 2026cond-mat.dis-nn cs.LG hep-latcited by 1

Not on the map. Why?

Abstract (arXiv)

The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open problem. We formulate neural network training as the stochastic evolution of an initially random matrix ensemble, driven by stochastic gradient descent (SGD) updates that reshape the spectral bulk while amplifying signal strength. This induces a Baik-Ben Arous-P'ech'e (BBP) transition during training, where isolated eigenvalues detach from the random bulk distribution, providing a dynamical framework for representation formation in high-dimensional learning dynamics. We demonstrate this in a solvable linear teacher-student model, where spectral evolution is analytically tractable and a phase diagram of trainability governed by the step size (or learning rate) and initial weight variance is obtained, and subsequently extend our formalism beyond the linear regime to nonlinear and stochastic settings. Numerical simulations in realistic settings support this picture, showing robust emergence of spectral alignment during training. Our results suggest that spectral analysis may provide a unified perspective of stochastic learning dynamics, linking trainability, optimisation hyperparameters, spectral phase transitions, and representation learning in neural networks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

38 of 66 references.

References · 38

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  5. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  6. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  7. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  9. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  10. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  11. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  12. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
26 more
  1. 2017A Random Matrix Approach to Neural Networkscited by 80
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  3. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  4. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  5. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  6. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  7. 2018Optimality and Sub-optimality of PCA I: Spiked Random Matrix Modelscited by 33
  8. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  9. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 25
  10. 2025The Universal Weight Subspace Hypothesiscited by 22
  11. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  12. 2023Spectral alignment of stochastic gradient descent for high-dimensional classification taskscited by 13
  13. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  14. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  15. 2014The local low-dimensionality of natural imagescited by 8
  16. 2024The Role of the Time-Dependent Hessian in High-Dimensional Optimizationcited by 7
  17. 2024Stochastic weight matrix dynamics during learning and Dyson Brownian motioncited by 7
  18. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 4
  19. 2025Local geometry of high-dimensional mixture models: Effective spectral theory and dynamical transitionscited by 3
  20. 2025Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networkscited by 2
  21. 2025Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networkscited by 2
  22. 2026On the Geometric Structure of Layer Updates in Deep Language Modelscited by 2
  23. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  24. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 2
  25. 2025Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networkscited by 1
  26. 2025Sparse corruption in low-rank matrix inference: the PCA benchmarkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Intruder Threshold: A Spectral Law for LoRA Fine-Tuningcited by 1