mlsanityData to 7 Oct 2026

Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

arXiv 1902.02880 ↗First posted 7 Feb 2019cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 25

Not on the map. Why?

Abstract (arXiv)

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 9; 2023: 5; 2024: 4; 2025: 0; 2026: 1.

25 papers.

Sort:

  1. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  2. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  3. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  4. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  5. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  6. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  7. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15
  8. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 15
  9. 2021Analysis of feature learning in weight-tied autoencoders via the mean field lenscited by 13
  10. 2022Two-Scale Gradient Descent Ascent Dynamics Finds Mixed Nash Equilibria of Continuous Games: A Mean-Field Perspectivecited by 13
  11. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
  12. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
13 more
  1. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
  2. 2022Provably convergent quasistatic dynamics for mean-field two-player zero-sum gamescited by 9
  3. 2022Learning and generalization of one-hidden-layer neural networks, going beyond standard Gaussian datacited by 8
  4. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  5. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  6. 2022A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networkscited by 3
  7. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  8. 2022Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channelscited by 2
  9. 2021A Riemannian Mean Field Formulation for Two-layer Neural Networks with Batch Normalizationcited by 1
  10. 2022Proximal Mean Field Learning in Shallow Neural Networkscited by 1
  11. 2024Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Inputcited by 1
  12. 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutionscited by 1
  13. 2024A Mean Field Ansatz for Zero-Shot Weight Transfercited by 0