# Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization arXiv: https://arxiv.org/abs/2110.08084 ยท HTML: https://mlsanity.com/paper/2110.08084/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 50 references that are in the map. - First posted on arXiv 15 Oct 2021. ## Builds on (most cited first, 25 of 25) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018) - [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017) - [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018) - [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018) - [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019) - [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014) - [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017) - [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017) - [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020) - [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018) - [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020) - [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019) - [Finite Depth and Width Corrections to the Neural Tangent Kernel](https://mlsanity.com/paper/1909.05989/) (2019) - [Variance-Reduced Methods for Machine Learning](https://mlsanity.com/paper/2010.00892/) (2020) - [A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks](https://mlsanity.com/paper/2001.11443/) (2020) - [Stochastic Particle Gradient Descent for Infinite Ensembles](https://mlsanity.com/paper/1712.05438/) (2017) - [Sparse Optimization on Measures with Over-parameterized Gradient Descent](https://mlsanity.com/paper/1907.10300/) (2019) - [A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network](https://mlsanity.com/paper/2102.02410/) (2021) - [Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/2007.01452/) (2020) - [Mean Field Analysis of Deep Neural Networks](https://mlsanity.com/paper/1903.04440/) (2019) - [On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime](https://mlsanity.com/paper/2005.13530/) (2020) - [On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics](https://mlsanity.com/paper/2007.15623/) (2020) - [On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting](https://mlsanity.com/paper/2106.06251/) (2021) - [A General Theory for Structured Prediction with Smooth Convex Surrogates](https://mlsanity.com/paper/1902.01958/) (2019) ## Built on by (most cited first, 16 of 16) - [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022) - [From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks](https://mlsanity.com/paper/2302.05882/) (2023) - [What can a Single Attention Layer Learn? A Study Through the Random Features Lens](https://mlsanity.com/paper/2307.11353/) (2023) - [Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey](https://mlsanity.com/paper/2305.05080/) (2023) - [Infinite-width limit of deep linear neural networks](https://mlsanity.com/paper/2211.16980/) (2022) - [Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD](https://mlsanity.com/paper/2305.18502/) (2023) - [Intrinsic dimensionality and generalization properties of the R-norm inductive bias](https://mlsanity.com/paper/2206.05317/) (2022) - [A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequality](https://mlsanity.com/paper/2311.15365/) (2023) - [Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime](https://mlsanity.com/paper/2504.18208/) (2025) - [Gradient flows on graphons: existence, convergence, continuity equations](https://mlsanity.com/paper/2111.09459/) (2021) - [FastPart: Over-Parameterized Stochastic Gradient Descent for Sparse optimisation on Measures](https://mlsanity.com/paper/2312.05993/) (2023) - [Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective](https://mlsanity.com/paper/2512.11784/) (2025) - [Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks](https://mlsanity.com/paper/2604.09412/) (2026) - [Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization](https://mlsanity.com/paper/2511.21466/) (2025) - [Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective](https://mlsanity.com/paper/2603.23831/) (2026) - [Fluctuations of Nonlinear Observables in Mean Field Neural Network Training](https://mlsanity.com/paper/2610.09768/) (2026) ## Abstract (arXiv) Many supervised machine learning methods are naturally cast as optimization problems. For prediction models which are linear in their parameters, this often leads to convex problems for which many mathematical guarantees exist. Models which are non-linear in their parameters such as neural networks lead to non-convex optimization problems for which guarantees are harder to obtain. In this review paper, we consider two-layer neural networks with homogeneous activation functions where the number of hidden neurons tends to infinity, and show how qualitative convergence guarantees may be derived.