mlsanityData to 7 Oct 2026

Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization

arXiv 2110.08084 ↗First posted 15 Oct 2021cs.LG math.OC math.STcited by 15

Not on the map. Why?

Abstract (arXiv)

Many supervised machine learning methods are naturally cast as optimization problems. For prediction models which are linear in their parameters, this often leads to convex problems for which many mathematical guarantees exist. Models which are non-linear in their parameters such as neural networks lead to non-convex optimization problems for which guarantees are harder to obtain. In this review paper, we consider two-layer neural networks with homogeneous activation functions where the number of hidden neurons tends to infinity, and show how qualitative convergence guarantees may be derived.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

25 of 50 references.

References · 25

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2017How to Escape Saddle Points Efficientlycited by 284
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
13 more
  1. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  2. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  3. 2020Variance-Reduced Methods for Machine Learningcited by 79
  4. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  5. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  6. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  7. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  8. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  9. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  10. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  11. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  12. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  13. 2019A General Theory for Structured Prediction with Smooth Convex Surrogatescited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 6; 2024: 0; 2025: 3; 2026: 2.

15 papers.

Sort:

  1. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  2. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  3. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  4. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  5. 2022Infinite-width limit of deep linear neural networkscited by 20
  6. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  7. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  8. 2023A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequalitycited by 4
  9. 2025Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regimecited by 4
  10. 2021Gradient flows on graphons: existence, convergence, continuity equationscited by 3
  11. 2023FastPart: Over-Parameterized Stochastic Gradient Descent for Sparse optimisation on Measurescited by 3
  12. 2025Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspectivecited by 3
3 more
  1. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  2. 2026Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspectivecited by 1
  3. 2025Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimizationcited by 0