mlsanityData to 7 Oct 2026

Stochastic weight matrix dynamics during learning and Dyson Brownian motion

arXiv 2407.16427 ↗First posted 23 Jul 2024cond-mat.dis-nn cs.LG hep-latcited by 7

Not on the map. Why?

Abstract (arXiv)

We demonstrate that the update of weight matrices in learning algorithms can be described in the framework of Dyson Brownian motion, thereby inheriting many features of random matrix theory. We relate the level of stochasticity to the ratio of the learning rate and the mini-batch size, providing more robust evidence to a previously conjectured scaling relationship. We discuss universal and non-universal features in the resulting Coulomb gas distribution and identify the Wigner surmise and Wigner semicircle explicitly in a teacher-student model and in the (near-)solvable case of the Gaussian restricted Boltzmann machine.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 11 of its 24 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

11 of 24 references.

References · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  4. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  5. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  6. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
  7. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  8. 2020Restricted Boltzmann Machine, recent advances and mean-field theorycited by 31
  9. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  10. 2021Appearance of Random Matrix Theory in Deep Learningcited by 7
  11. 2018Stochastic natural gradient descent draws posterior samples in function spacecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024On learning higher-order cumulants in diffusion modelscited by 10
  2. 2025From SGD to Spectra: A Theory of Neural Network Weight Dynamicscited by 8
  3. 2025Physics-Driven Learning for Inverse Problems in Quantum Chromodynamicscited by 7
  4. 2026Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkincited by 2
  5. 2025Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networkscited by 1
  6. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 1
  7. 2024Random Matrix Theory for Stochastic Gradient Descentcited by 0