# Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

arXiv: https://arxiv.org/abs/2308.08977 · HTML: https://mlsanity.com/paper/2308.08977/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 53 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 7 of 7)

- [Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition](https://mlsanity.com/paper/1308.6370/) (2013)
- [Second-order regression models exhibit progressive sharpening to the edge of stability](https://mlsanity.com/paper/2210.04860/) (2022)
- [The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime](https://mlsanity.com/paper/1911.01544/) (2019)
- [Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties](https://mlsanity.com/paper/2205.07069/) (2022)
- [Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions](https://mlsanity.com/paper/2206.07252/) (2022)
- [Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD](https://mlsanity.com/paper/2305.18502/) (2023)
- [Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA](https://mlsanity.com/paper/1712.04332/) (2017)

## Built on by (most cited first, 26 of 26)

- [Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit](https://mlsanity.com/paper/2406.01581/) (2024)
- [Gradient-Based Feature Learning under Structured Data](https://mlsanity.com/paper/2309.03843/) (2023)
- [Bayes-optimal learning of an extensive-width neural network from quadratically many samples](https://mlsanity.com/paper/2408.03733/) (2024)
- [Fundamental computational limits of weak learnability in high-dimensional multi-index models](https://mlsanity.com/paper/2405.15480/) (2024)
- [High dimensional analysis reveals conservative sharpening and a stochastic edge of stability](https://mlsanity.com/paper/2404.19261/) (2024)
- [Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics](https://mlsanity.com/paper/2408.07254/) (2024)
- [A theoretical perspective on mode collapse in variational inference](https://mlsanity.com/paper/2410.13300/) (2024)
- [Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks](https://mlsanity.com/paper/2506.02651/) (2025)
- [Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery](https://mlsanity.com/paper/2502.01583/) (2025)
- [To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions](https://mlsanity.com/paper/2406.11733/) (2024)
- [Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval](https://mlsanity.com/paper/2601.22652/) (2026)
- [Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning](https://mlsanity.com/paper/2506.01656/) (2025)
- [The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms](https://mlsanity.com/paper/2405.19585/) (2024)
- [Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs](https://mlsanity.com/paper/2406.02157/) (2024)
- [Bayes optimal learning of attention-indexed models](https://mlsanity.com/paper/2506.01582/) (2025)
- [Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data](https://mlsanity.com/paper/2511.18661/) (2025)
- [Universality of high-dimensional scaling limits of stochastic gradient descent](https://mlsanity.com/paper/2512.13634/) (2025)
- [Phases of Muon: When Muon Eclipses SignSGD](https://mlsanity.com/paper/2605.09552/) (2026)
- [SGD with memory: fundamental properties and stochastic acceleration](https://mlsanity.com/paper/2410.04228/) (2024)
- [A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models](https://mlsanity.com/paper/2603.17896/) (2026)
- [Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks](https://mlsanity.com/paper/2511.02258/) (2025)
- [Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval](https://mlsanity.com/paper/2602.17779/) (2026)
- [High dimensional theory of two-phase optimizers](https://mlsanity.com/paper/2603.26954/) (2026)
- [Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networks](https://mlsanity.com/paper/2604.03068/) (2026)
- [When Stronger Triggers Backfire: A High-Dimensional Theory of Backdoor Attacks](https://mlsanity.com/paper/2605.22481/) (2026)
- [Large-scale Uncertainty Quantification for Latent Variable Models Using Subsampling Markov Chain Monte Carlo](https://mlsanity.com/paper/2606.00309/) (2026)

## Abstract (arXiv)

We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance. In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality. This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data. This framework allows us to obtain learning rate thresholds for stability of SGD as well as convergence guarantees. In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD) which allows us to analyze the dynamics of general statistics of SGD iterates. Finally, we illustrate this theory on some standard examples and show numerical simulations which give an excellent match to the theory.