Random Matrix Theory for Stochastic Gradient Descent
Not on the map. Why?
Abstract (arXiv)
Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here we apply concepts from random matrix theory to describe stochastic weight matrix dynamics, using the framework of Dyson Brownian motion. We derive the linear scaling rule between the learning rate (step size) and the batch size, and identify universal and non-universal aspects of weight matrix dynamics. We test our findings in the (near-)solvable case of the Gaussian Restricted Boltzmann Machine and in a linear one-hidden-layer neural network.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 26 references that are in the map. First posted on arXiv 29 Dec 2024.
Builds on
11 of 26 references.
References · 11
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
- 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
- 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 48
- 2020Restricted Boltzmann Machine, recent advances and mean-field theorycited by 31
- 2021Appearance of Random Matrix Theory in Deep Learningcited by 7
- 2024Stochastic weight matrix dynamics during learning and Dyson Brownian motioncited by 7
- 2018Stochastic natural gradient descent draws posterior samples in function spacecited by 4
- 2024Dyson Brownian motion and random matrix dynamics of weight matrices during learningcited by 3
Built on by
Not cited yet.