mlsanityData to 7 Oct 2026

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

arXiv 2202.00293 ↗First posted 1 Feb 2022stat.ML cond-mat.dis-nn cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow networks, which tend to get stuck in badly-generalizing local minima. Here we investigate the cross-over between these two regimes in the high-dimensional setting, and in particular investigate the connection between the so-called mean-field/hydrodynamic regime and the seminal approach of Saad & Solla. Focusing on the case of Gaussian data, we study the interplay between the learning rate, the time scale, and the number of hidden units in the high-dimensional dynamics of stochastic gradient descent (SGD). Our work builds on a deterministic description of SGD in high-dimensions from statistical physics, which we extend and for which we provide rigorous convergence rates.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

16 of 34 references.

References · 16

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  6. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  9. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  10. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  11. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  12. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
4 more
  1. 2017Phase Retrieval via Randomized Kaczmarz: Theoretical Guaranteescited by 15
  2. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
  3. 2018A Solvable High-Dimensional Model of GANcited by 13
  4. 2018Subspace Estimation from Incomplete Observations: A High-Dimensional Analysiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 15; 2024: 8; 2025: 13; 2026: 9.

50 papers.

Sort:

By area:

  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  3. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  4. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  5. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  6. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  7. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  8. 2023Mapping of attention mechanisms to a generalized Potts modelcited by 19
  9. 2023Leveraging the two timescale regime to demonstrate convergence of neural networkscited by 18
  10. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  11. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  12. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
38 more
  1. 2024A theoretical perspective on mode collapse in variational inferencecited by 14
  2. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  3. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  4. 2023Forward Learning with Top-Down Feedback: Empirical and Analytical Characterizationcited by 13
  5. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  6. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  7. 2023Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealingcited by 12
  8. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  9. 2022Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energycited by 9
  10. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networkscited by 9
  11. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  12. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  13. 2026A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalizationcited by 7
  14. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 6
  15. 2023High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariancecited by 5
  16. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  17. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 5
  18. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  19. 2023Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscapecited by 3
  20. 2025Bayes optimal learning of attention-indexed modelscited by 3
  21. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  22. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 2
  23. 2024Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learningcited by 2
  24. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  25. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  26. 2026Statistical Properties of Training & Generalizationcited by 2
  27. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  28. 2024Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limitcited by 1
  29. 2025Joint Learning in the Gaussian Single Index Modelcited by 1
  30. 2025Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networkscited by 1
  31. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  32. 2023Minibatch training of neural network ensembles via trajectory samplingcited by 0
  33. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
  34. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0
  35. 2025On the Effect of Regularization on Nonparametric Mean-Variance Regressioncited by 0
  36. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 0
  37. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 0
  38. 2026Singular perturbations and hierarchical learning in two-layer neural networkscited by 0