# Stochastic weight matrix dynamics during learning and Dyson Brownian motion

arXiv: https://arxiv.org/abs/2407.16427 · HTML: https://mlsanity.com/paper/2407.16427/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)
- [Fluctuation-dissipation relations for stochastic gradient descent](https://mlsanity.com/paper/1810.00004/) (2018)
- [Restricted Boltzmann Machine, recent advances and mean-field theory](https://mlsanity.com/paper/2011.11307/) (2020)
- [On the different regimes of Stochastic Gradient Descent](https://mlsanity.com/paper/2309.10688/) (2023)
- [Appearance of Random Matrix Theory in Deep Learning](https://mlsanity.com/paper/2102.06740/) (2021)
- [Stochastic natural gradient descent draws posterior samples in function space](https://mlsanity.com/paper/1806.09597/) (2018)

## Built on by (most cited first, 7 of 7)

- [On learning higher-order cumulants in diffusion models](https://mlsanity.com/paper/2410.21212/) (2024)
- [From SGD to Spectra: A Theory of Neural Network Weight Dynamics](https://mlsanity.com/paper/2507.12709/) (2025)
- [Physics-Driven Learning for Inverse Problems in Quantum Chromodynamics](https://mlsanity.com/paper/2501.05580/) (2025)
- [Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin](https://mlsanity.com/paper/2604.13123/) (2026)
- [Spectral phase transitions and trainability in neural network learning dynamics](https://mlsanity.com/paper/2606.28486/) (2026)
- [Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks](https://mlsanity.com/paper/2509.01349/) (2025)
- [Random Matrix Theory for Stochastic Gradient Descent](https://mlsanity.com/paper/2412.20496/) (2024)

## Abstract (arXiv)

We demonstrate that the update of weight matrices in learning algorithms can be described in the framework of Dyson Brownian motion, thereby inheriting many features of random matrix theory. We relate the level of stochasticity to the ratio of the learning rate and the mini-batch size, providing more robust evidence to a previously conjectured scaling relationship. We discuss universal and non-universal features in the resulting Coulomb gas distribution and identify the Wigner surmise and Wigner semicircle explicitly in a teacher-student model and in the (near-)solvable case of the Gaussian restricted Boltzmann machine.