mlsanityData to 7 Oct 2026

High-dimensional limit theorems for SGD: Effective dynamics and critical scaling

arXiv 2206.04030 ↗First posted 8 Jun 2022stat.ML cs.LG math.PRcited by 69

Not on the map. Why?

Abstract (arXiv)

We study the scaling limits of stochastic gradient descent (SGD) with constant step-size in the high-dimensional regime. We prove limit theorems for the trajectories of summary statistics (i.e., finite-dimensional functions) of SGD as the dimension goes to infinity. Our approach allows one to choose the summary statistics that are tracked, the initialization, and the step-size. It yields both ballistic (ODE) and diffusive (SDE) limits, with the limit depending dramatically on the former choices. We show a critical scaling regime for the step-size, below which the effective ballistic dynamics matches gradient flow for the population loss, but at which, a new correction term appears which changes the phase diagram. About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate. We demonstrate our approach on popular examples including estimation for spiked matrix and tensor models and classification via two-layer networks for binary and XOR-type Gaussian mixture models. These examples exhibit surprising phenomena including multimodal timescales to convergence as well as convergence to sub-optimal solutions with probability bounded away from zero from random (e.g., Gaussian) initializations. At the same time, we demonstrate the benefit of overparametrization by showing that the latter probability goes to zero as the second layer width grows.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 39 of its 87 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

39 of 87 references.

References · 39

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  5. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  6. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  7. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  8. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  9. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  11. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  12. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
27 more
  1. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  2. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  3. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 94
  4. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 84
  5. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  6. 2014A statistical model for tensor PCAcited by 72
  7. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
  8. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  9. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  10. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  11. 2015Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectorscited by 38
  12. 2015Tensor principal component analysis via sum-of-squares proofscited by 35
  13. 2017The landscape of the spiked tensor modelcited by 34
  14. 2018Optimality and Sub-optimality of PCA I: Spiked Random Matrix Modelscited by 33
  15. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  16. 2019Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLTcited by 30
  17. 2016Statistical limits of spiked tensor modelscited by 27
  18. 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
  19. 2018Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inferencecited by 24
  20. 2018Fundamental limits of detection in the spiked Wigner modelcited by 18
  21. 2015Convergence of Stochastic Gradient Descent for PCAcited by 17
  22. 2017Community Detection in Hypergraphs, Spiked Tensor Models, and Sum-of-Squarescited by 17
  23. 2017Phase Retrieval via Randomized Kaczmarz: Theoretical Guaranteescited by 15
  24. 2017Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCAcited by 11
  25. 2018Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processescited by 6
  26. 2018Diffusion Approximations for Online Principal Component Estimation and Global Convergencecited by 3
  27. 2022High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 18; 2024: 23; 2025: 11; 2026: 16.

69 papers.

Sort:

By area:

  1. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  3. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  4. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  5. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  6. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  7. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  8. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  9. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  10. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  11. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  12. 2023On Single Index Models beyond Gaussian Datacited by 17
38 more
  1. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 15
  2. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  3. 2024A theoretical perspective on mode collapse in variational inferencecited by 14
  4. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  5. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  6. 2024Shallow diffusion networks provably learn hidden low-dimensional structurecited by 12
  7. 2023The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasetscited by 9
  8. 2024Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networkscited by 9
  9. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  10. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  11. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  12. 2024The Role of the Time-Dependent Hessian in High-Dimensional Optimizationcited by 7
  13. 2026A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalizationcited by 7
  14. 2025A Theory of Initialisation's Impact on Specialisationcited by 6
  15. 2023High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariancecited by 5
  16. 2023Symmetric Single Index Learningcited by 5
  17. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 5
  18. 2024Hyperparameter tuning via trajectory predictions: Stochastic prox-linear methods in matrix sensingcited by 4
  19. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  20. 2023Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscapecited by 3
  21. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  22. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  23. 2024Limit Theorems for Stochastic Gradient Descent with Infinite Variancecited by 3
  24. 2025Geometry of Neural Reinforcement Learning in Continuous State and Action Spacescited by 3
  25. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  26. 2024Fundamental limits of Non-Linear Low-Rank Matrix Estimationcited by 2
  27. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 2
  28. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  29. 2025Generalization Bounds for Markov Algorithms through Entropy Flow Computationscited by 2
  30. 2025EMA Without the Lag: Bias-Corrected Iterate Averaging Schemescited by 2
  31. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  32. 2023Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regressioncited by 1
  33. 2023Gradient flow on extensive-rank positive semi-definite matrix denoisingcited by 1
  34. 2023Hotelling Deflation on Large Symmetric Spiked Tensorscited by 1
  35. 2023A Nested Matrix-Tensor Model for Noisy Multi-view Clusteringcited by 1
  36. 2023Beyond Log-Concavity: Theory and Algorithm for Sum-Log-Concave Optimizationcited by 1
  37. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  38. 2024Faster Convergence for Transformer Fine-tuning with Line Search Methodscited by 1

50 most cited of 69 All 69 (JSON)