# The Dynamics of Learning: A Random Matrix Approach

arXiv: https://arxiv.org/abs/1805.11917 · HTML: https://mlsanity.com/paper/1805.11917/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2018.

## Built on by (most cited first, 15 of 15)

- [Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions](https://mlsanity.com/paper/2206.01029/) (2022)
- [Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model](https://mlsanity.com/paper/2110.11805/) (2021)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [A Random Matrix Perspective on Random Tensors](https://mlsanity.com/paper/2108.00774/) (2021)
- [SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning](https://mlsanity.com/paper/2401.03137/) (2024)
- [Two-way kernel matrix puncturing: towards resource-efficient PCA and spectral clustering](https://mlsanity.com/paper/2102.12293/) (2021)
- [Random matrices in service of ML footprint: ternary random features with no performance loss](https://mlsanity.com/paper/2110.01899/) (2021)
- [Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures](https://mlsanity.com/paper/2402.02697/) (2024)
- [High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile](https://mlsanity.com/paper/2403.20200/) (2024)
- [Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models](https://mlsanity.com/paper/2506.13139/) (2025)
- ["Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach](https://mlsanity.com/paper/2403.00258/) (2024)
- [Deep Regression Ensembles](https://mlsanity.com/paper/2203.05417/) (2022)
- [A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention](https://mlsanity.com/paper/2507.09394/) (2025)
- [SnatchML: Hijacking ML models without Training Access](https://mlsanity.com/paper/2406.01708/) (2024)
- [A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-norm](https://mlsanity.com/paper/2608.24210/) (2026)

## Abstract (arXiv)

Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neural nets work so well today. In this paper, we introduce a random matrix-based framework to analyze the learning dynamics of a single-layer linear network on a binary classification problem, for data of simultaneously large dimension and size, trained by gradient descent. Our results provide rich insights into common questions in neural nets, such as overfitting, early stopping and the initialization of training, thereby opening the door for future studies of more elaborate structures and models appearing in today's neural networks.