mlsanityData to 7 Oct 2026

On Feature Learning in Neural Networks with Global Convergence Guarantees

arXiv 2204.10782 ↗First posted 22 Apr 2022cs.LG math.OC math.PRcited by 12

Not on the map. Why?

Abstract (arXiv)

We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that allow feature learning while admitting non-asymptotic global convergence guarantees. First, for wide shallow NNs under the mean-field scaling and with a general class of activation functions, we prove that when the input dimension is no less than the size of the training set, the training loss converges to zero at a linear rate under GF. Building upon this analysis, we study a model of wide multi-layer NNs whose second-to-last layer is trained via GF, for which we also prove a linear-rate convergence of the training loss to zero, but regardless of the input dimension. We also show empirically that, unlike in the Neural Tangent Kernel (NTK) regime, our multi-layer model exhibits feature learning and can achieve better generalization performance than its NTK counterpart.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 60 of its 74 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

60 of 74 references.

References · 60

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
38 more
  1. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  2. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  3. 2016Deep Learning without Poor Local Minimacited by 233
  4. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  5. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  6. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  7. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  10. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  11. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  12. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  13. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  14. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  15. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  16. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  17. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  18. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  19. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  20. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  21. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  22. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  23. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  24. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  25. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  26. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  27. 2019Machine Learning from a Continuous Viewpointcited by 50
  28. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  29. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  30. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  31. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  32. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  33. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  34. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  35. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  36. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  37. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  38. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 2; 2025: 5.

12 papers.

  1. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  2. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  3. 2023Rethinking Gauss-Newton for learning over-parameterized modelscited by 6
  4. 2025A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Modelscited by 4
  5. 2023Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learningcited by 3
  6. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  7. 2022Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Ratescited by 2
  8. 2025Convergence of Shallow ReLU Networks on Weakly Interacting Datacited by 2
  9. 2025Curse of Dimensionality in Neural Network Optimizationcited by 1
  10. 2024Review-Based Hyperbolic Cross-Domain Recommendationcited by 0
  11. 2025Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regimecited by 0
  12. 2025Gradient Flow Convergence Guarantee for General Neural Network Architecturescited by 0