Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Rates
Not on the map. Why?
Abstract (arXiv)
We consider optimizing two-layer neural networks in the mean-field regime where the learning dynamics of network weights can be approximated by the evolution in the space of probability measures over the weight parameters associated with the neurons. The mean-field regime is a theoretically attractive alternative to the NTK (lazy training) regime which is only restricted locally in the so-called neural tangent kernel space around specialized initializations. Several prior works (citechizat2018global, mei2018mean) establish the asymptotic global optimality of the mean-field regime, but it is still challenging to obtain a quantitative convergence rate due to the complicated unbounded nonlinearity of the training dynamics. This work establishes the first linear convergence result for vanilla two-layer neural networks trained by continuous-time noisy gradient descent in the mean-field regime. Our result relies on a novel time-depdendent estimate of the logarithmic Sobolev constants for a family of measures determined by the evolving distribution of hidden neurons.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 19 May 2022.
Builds on
24 of 47 references.
References · 24
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
- 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
12 more
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2017Stein Variational Gradient Descent as Gradient Flowcited by 149
- 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
- 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
- 2018Sampling Can Be Faster Than Optimizationcited by 68
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2020SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergencecited by 64
- 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
- 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
- 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
- 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
- 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1.
2 papers.