mlsanityData to 7 Oct 2026

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

arXiv 2308.08977 ↗First posted 17 Aug 2023math.OC cs.LG math.PRcited by 25

Not on the map. Why?

Abstract (arXiv)

We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance. In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality. This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data. This framework allows us to obtain learning rate thresholds for stability of SGD as well as convergence guarantees. In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD) which allows us to analyze the dynamics of general statistics of SGD iterates. Finally, we illustrate this theory on some standard examples and show numerical simulations which give an excellent match to the theory.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 7 of its 53 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

7 of 53 references.

References · 7

  1. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  2. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  3. 2019The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regimecited by 27
  4. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  5. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  6. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  7. 2017Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCAcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 7; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  2. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  3. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  4. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  5. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 15
  6. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 14
  7. 2024A theoretical perspective on mode collapse in variational inferencecited by 14
  8. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  9. 2025Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recoverycited by 6
  10. 2024To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensionscited by 5
  11. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 5
  12. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learningcited by 4
13 more
  1. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  2. 2025Bayes optimal learning of attention-indexed modelscited by 3
  3. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  4. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 2
  5. 2024SGD with memory: fundamental properties and stochastic accelerationcited by 2
  6. 2025Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networkscited by 1
  7. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  8. 2026High dimensional theory of two-phase optimizerscited by 1
  9. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 0
  10. 2026Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrievalcited by 0
  11. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 0
  12. 2026When Stronger Triggers Backfire: A High-Dimensional Theory of Backdoor Attackscited by 0
  13. 2026Large-scale Uncertainty Quantification for Latent Variable Models Using Subsampling Markov Chain Monte Carlocited by 0