mlsanityData to 7 Oct 2026

A Mean Field View of the Landscape of Two-Layers Neural Networks

arXiv 1804.06561 ↗First posted 18 Apr 2018stat.ML cond-mat.stat-mech cs.LGcited by 463

Not on the map. Why?

Abstract (arXiv)

Multi-layer neural networks are among the most powerful models in machine learning, yet the fundamental reasons for this success defy mathematical understanding. Learning a neural network requires to optimize a non-convex high-dimensional objective (risk function), a problem which is usually attacked using stochastic gradient descent (SGD). Does SGD converge to a global optimum of the risk or only to a local optimum? In the first case, does this happen because local minima are absent, or because SGD somehow avoids them? In the second, why do local minima reached by SGD have good generalization properties? In this paper we consider a simple case, namely two-layers neural networks, and prove that -in a suitable scaling limit- SGD dynamics is captured by a certain non-linear partial differential equation (PDE) that we call distributional dynamics (DD). We then consider several specific examples, and show how DD can be used to prove convergence of SGD to networks with nearly ideal generalization error. This description allows to 'average-out' some of the complexities of the landscape of neural networks, and can be used to prove a general convergence result for noisy SGD.

Key facts. Cited by 463 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 92; 2022: 72; 2023: 79; 2024: 81; 2025: 70; 2026: 69.

463 papers.

Sort:

By area:

  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  3. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  4. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  5. 2023A mathematical perspective on Transformerscited by 160
  6. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  7. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  8. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  9. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  10. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  12. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
38 more
  1. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  2. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  3. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  4. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  5. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  6. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  7. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  8. 2023Grokking modular arithmeticcited by 77
  9. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  10. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  11. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  12. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  13. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  14. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  15. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  16. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  17. 2023Learning time-scales in two-layers neural networkscited by 57
  18. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  19. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  20. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  21. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  22. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  23. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  24. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  25. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  26. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  27. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  28. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  29. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  30. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  31. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  32. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  33. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  34. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  35. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  36. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  37. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  38. 2021KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Supportcited by 37

50 most cited of 463 All 463 (JSON)