mlsanityData to 14 Oct 2026

Mean Field Analysis of Deep Neural Networks

arXiv 1903.04440 ↗First posted 11 Mar 2019math.PR stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

We analyze multi-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multi-layer neural network output. The limit procedure is valid for any number of hidden layers and it naturally also describes the limiting behavior of the training loss. The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization. The limit satisfies a system of deterministic integro-differential equations. The proof uses methods from weak convergence and stochastic analysis. We show that, under suitable assumptions on the activation functions and the behavior for large times, the limit neural network recovers a global minimum (with zero loss for the objective function).

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 13; 2024: 5; 2025: 1; 2026: 4.

42 papers.

Sort:

By area:

  1. 2023A Spectral Condition for Feature Learningcited by 109
  2. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  3. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 35
  4. 2023Generalization Ability of Wide Neural Networks on ℝcited by 30
  5. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  6. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  7. 2023When can transformers reason with abstract symbols?cited by 24
  8. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  9. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  10. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  11. 2023Quantitative CLTs in Deep Neural Networkscited by 20
  12. 2023A Rainbow in Deep Network Black Boxescited by 18
30 more
  1. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  2. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 16
  3. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  4. 2022Two-Scale Gradient Descent Ascent Dynamics Finds Mixed Nash Equilibria of Continuous Games: A Mean-Field Perspectivecited by 13
  5. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 12
  6. 2024Shallow diffusion networks provably learn hidden low-dimensional structurecited by 12
  7. 2022Provably convergent quasistatic dynamics for mean-field two-player zero-sum gamescited by 9
  8. 2022On the generalization of learning algorithms that do not convergecited by 9
  9. 2023Depth Separation with Multilayer Mean-Field Networkscited by 9
  10. 2022Normalization effects on deep neural networkscited by 6
  11. 2023High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariancecited by 6
  12. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  13. 2023Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGDcited by 5
  14. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  15. 2024Generalization Error of Graph Neural Networks in the Mean-field Regimecited by 4
  16. 2022A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networkscited by 3
  17. 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Netscited by 3
  18. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  19. 2022Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channelscited by 2
  20. 2022Proximal Mean Field Learning in Shallow Neural Networkscited by 1
  21. 2023Global Optimality of Elman-type RNN in the Mean-Field Regimecited by 1
  22. 2023Tight conditions for when the NTK approximation is validcited by 1
  23. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  24. 2024Understanding Transfer Learning via Mean-field Analysiscited by 1
  25. 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutionscited by 1
  26. 2024A Mean Field Ansatz for Zero-Shot Weight Transfercited by 0
  27. 2025Mean-field limit from general mixtures of experts to quantum neural networkscited by 0
  28. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
  29. 2026A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networkscited by 0
  30. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0