A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
Not on the map. Why?
Abstract (arXiv)
To understand the training dynamics of neural networks, prior studies have considered the mean-field limit of two-layer neural networks as the width tends to infinity, establishing theoretical guarantees for its convergence under gradient flow training as well as approximation and generalization capabilities. In this work, we study the infinite-width limit of a type of three-layer neural network where the first-layer weights are randomly sampled and untrained. To rigorously define the limiting model, we extend the mean-field theory by lifting the representation of neurons from Euclidean to functional spaces. This allows us to establish the mean-field training dynamics as a functional gradient flow with a time-varying kernel that remains positive-definite under suitable assumptions, thus proving a linear-rate convergence of its training loss. Furthermore, we define novel function spaces that contain the solutions obtained through the mean-field training dynamics and prove Rademacher complexity bounds for these spaces. Notably, our analysis applies to a range of scaling choices of the model, resulting in two distinct regimes of the mean-field limit that both exhibit feature learning through training.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 51 of its 80 references that are in the map. First posted on arXiv 28 Oct 2022.
Builds on
51 of 80 references.
References · 51
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
38 more
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
- 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2015The Power of Depth for Feedforward Neural Networkscited by 181
- 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
- 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
- 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
- 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
- 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
- 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
- 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
- 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
- 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
- 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
- 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
- 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
- 2019Mean Field Analysis of Deep Neural Networkscited by 42
- 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
- 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
- 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
- 2021Two-layer neural networks with values in a Banach spacecited by 26
- 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
- 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
- 2020Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspectivecited by 22
- 2020Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysiscited by 21
- 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
- 2021Optimization-Based Separations for Neural Networkscited by 17
- 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
- 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2023: 3.
3 papers.