mlsanityData to 7 Oct 2026

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

arXiv 1902.06015 ↗First posted 16 Feb 2019stat.ML cond-mat.stat-mech cs.LGcited by 187

Not on the map. Why?

Abstract (arXiv)

We consider learning two layer neural networks using stochastic gradient descent. The mean-field description of this learning dynamics approximates the evolution of the network weights by an evolution in the space of probability distributions in RD (where D is the number of parameters associated to each neuron). This evolution can be defined through a partial differential equation or, equivalently, as the gradient flow in the Wasserstein space of probability distributions. Earlier work shows that (under some regularity assumptions), the mean field description is accurate as soon as the number of hidden units is much larger than the dimension D. In this paper we establish stronger and more general approximation guarantees. First of all, we show that the number of hidden units only needs to be larger than a quantity dependent on the regularity properties of the data, and independent of the dimensions. Next, we generalize this analysis to the case of unbounded activation functions, which was not covered by earlier bounds. We extend our results to noisy stochastic gradient descent. Finally, we show that kernel ridge regression can be recovered as a special limit of the mean field analysis.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 29; 2023: 41; 2024: 28; 2025: 29; 2026: 23.

187 papers.

Sort:

By area:

  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  3. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  4. 2023A Spectral Condition for Feature Learningcited by 101
  5. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  6. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 78
  7. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  8. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  9. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  10. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  11. 2023Learning time-scales in two-layers neural networkscited by 57
  12. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
38 more
  1. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  2. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  3. 2022Learning threshold neurons via the "edge of stability"cited by 50
  4. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  5. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  6. 2021Implicit Regularization in Tensor Factorizationcited by 43
  7. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  8. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  9. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  10. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  11. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  12. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  13. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  14. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  15. 2022Trajectory Inference via Mean-field Langevin in Path Spacecited by 29
  16. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  17. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  18. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  19. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  20. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  21. 2023When can transformers reason with abstract symbols?cited by 23
  22. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  23. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  24. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  25. 2023Variational Annealing on Graphs for Combinatorial Optimizationcited by 21
  26. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  27. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 20
  28. 2022Infinite-width limit of deep linear neural networkscited by 20
  29. 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 20
  30. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  31. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  32. 2023Six Lectures on Linearized Neural Networkscited by 19
  33. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
  34. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
  35. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  36. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  37. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  38. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17

50 most cited of 187 All 187 (JSON)