mlsanityData to 7 Oct 2026

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

arXiv 2006.06098 ↗First posted 10 Jun 2020cs.LG cond-mat.dis-nn math.STcited by 60

Not on the map. Why?

Abstract (arXiv)

We analyze in a closed form the learning dynamics of stochastic gradient descent (SGD) for a single-layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of a non-convex loss landscape with interpolating regimes and a large generalization gap. We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow. In the full-batch limit, we recover the standard gradient flow. We apply dynamical mean-field theory from statistical physics to track the dynamics of the algorithm in the high-dimensional limit via a self-consistent stochastic process. We explore the performance of the algorithm as a function of the control parameters shedding light on how it navigates the loss landscape.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 10; 2023: 10; 2024: 11; 2025: 11; 2026: 11.

60 papers.

Sort:

By area:

  1. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  2. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  3. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  4. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  5. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  6. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  7. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  8. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  9. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  10. 2021Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcited by 23
  11. 2025Generalization through variance: how noise shapes inductive biases in diffusion modelscited by 23
  12. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
38 more
  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  2. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  3. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  4. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
  5. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
  6. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  7. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  8. 2024High-dimensional learning of narrow neural networkscited by 12
  9. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 10
  10. 2022Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energycited by 9
  11. 2022Learning and generalization of one-hidden-layer neural networks, going beyond standard Gaussian datacited by 8
  12. 2023Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamicscited by 6
  13. 2024How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalancecited by 6
  14. 2021Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problemscited by 5
  15. 2022Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structurescited by 5
  16. 2022Optimal learning rate schedules in high-dimensional non-convex optimization problemscited by 4
  17. 2023Meta-Learning Strategies through Value Maximization in Neural Networkscited by 4
  18. 2023Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscapecited by 3
  19. 2023Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problemscited by 3
  20. 2024Asymptotic Dynamics of Alternating Minimization for Bilinear Regressioncited by 3
  21. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  22. 2024Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokenscited by 3
  23. 2025A statistical physics framework for optimal learningcited by 3
  24. 2025A Proof of Learning Rate Transfer under μPcited by 3
  25. 2021Exact solution to the random sequential dynamics of a message passing algorithmcited by 2
  26. 2022High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Modelscited by 2
  27. 2025Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networkscited by 2
  28. 2025Analytic theory of dropout regularizationcited by 2
  29. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 2
  30. 2026Statistical Properties of Training & Generalizationcited by 2
  31. 2022Analysis of Random Sequential Message Passing Algorithms for Approximate Inferencecited by 1
  32. 2022The Underlying Correlated Dynamics in Neural Trainingcited by 1
  33. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  34. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  35. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  36. 2026Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networkscited by 1
  37. 2021Learning Curves for SGD on Structured Featurescited by 0
  38. 2023Loss Dynamics of Temporal Difference Reinforcement Learningcited by 0

50 most cited of 60 All 60 (JSON)