# High-dimensional Limit of SGD for Diagonal Linear Networks

arXiv: https://arxiv.org/abs/2605.17177 · HTML: https://mlsanity.com/paper/2605.17177/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 59 references that are in the map.
- First posted on arXiv 16 May 2026.

## Builds on (most cited first, 2 of 2)

- [Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA](https://mlsanity.com/paper/1712.04332/) (2017)
- [On progressive sharpening, flat minima and generalisation](https://mlsanity.com/paper/2305.14683/) (2023)

## Abstract (arXiv)

Understanding the behavior of stochastic gradient methods is a central problem in modern machine learning. Recent work has highlighted diagonal linear networks as a simplified yet expressive setting for analyzing the optimization and generalization properties of neural models. In this work, we show that in the high-dimensional regime, stochastic gradient descent on diagonal linear networks is well-approximated by continuous dynamics governed by a stochastic differential equation (SDE), which explicitly decouples the drift from the gradient noise. We further derive a deterministic partial differential equation whose solution propagates the relevant state of the iterates and characterizes the time evolution of a broad class of observable statistics, including the risk, curvature, and other metrics for optimality. Finally, we show that, under a suitable parametrization, the stochastic dynamics are globally well posed and converge exponentially fast to zero risk with high probability, yielding a fully explicit non-asymptotic description of their long-time behavior. Numerical simulations corroborate our theoretical findings.