mlsanityData to 7 Oct 2026

From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks

arXiv 2302.05882 ↗First posted 12 Feb 2023stat.ML cond-mat.dis-nn cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a two-layer neural network trained on Gaussian data and labels generated by a similar, though not necessarily identical, target function. We rigorously analyse the limiting dynamics via a deterministic and low-dimensional description in terms of the sufficient statistics for the population risk. Our unifying analysis bridges different regimes of interest, such as the classical gradient-flow regime of vanishing learning rate, the high-dimensional regime of large input dimension, and the overparameterised "mean-field" regime of large network width, covering as well the intermediate regimes where the limiting dynamics is determined by the interplay between these behaviours. In particular, in the high-dimensional limit, the infinite-width dynamics is found to remain close to a low-dimensional subspace spanned by the target principal directions. Our results therefore provide a unifying picture of the limiting SGD dynamics with synthetic data.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 12 Feb 2023.

Builds on

22 of 42 references.

References · 22

  1. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  4. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  5. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  6. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  7. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  8. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  9. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  10. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  11. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 84
  12. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
10 more
  1. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  2. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  3. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  4. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  5. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  6. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 41
  7. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  8. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
  9. 2022On the symmetries in the dynamics of wide two-layer neural networkscited by 8
  10. 2022On Uniform Boundedness Properties of SGD and its Momentum Variantscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 12; 2025: 9; 2026: 7.

42 papers.

Sort:

  1. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  3. 2023Learning time-scales in two-layers neural networkscited by 57
  4. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  5. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  6. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  7. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  8. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
  9. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  10. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  11. 2023On Single Index Models beyond Gaussian Datacited by 17
  12. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
30 more
  1. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 15
  2. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  3. 2024A theoretical perspective on mode collapse in variational inferencecited by 14
  4. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  5. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  6. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  7. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  8. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 6
  9. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  10. 2023Symmetric Single Index Learningcited by 5
  11. 2025A theoretical framework for overfitting in energy-based modelingcited by 5
  12. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  13. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 5
  14. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4
  15. 2025Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexitycited by 4
  16. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  17. 2025Bayes optimal learning of attention-indexed modelscited by 3
  18. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  19. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 2
  20. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  21. 2024Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learningcited by 2
  22. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  23. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  24. 2026Statistical Properties of Training & Generalizationcited by 2
  25. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  26. 2026Interactive Learning of Single-Index Models via Stochastic Gradient Descentcited by 1
  27. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  28. 2024Error dynamics of mini-batch gradient descent with random reshuffling for least squares regressioncited by 0
  29. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 0
  30. 2026Singular perturbations and hierarchical learning in two-layer neural networkscited by 0