# On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime

arXiv: https://arxiv.org/abs/2110.02926 · HTML: https://mlsanity.com/paper/2110.02926/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 30 of 32)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data](https://mlsanity.com/paper/1808.01204/) (2018)
- [Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://mlsanity.com/paper/1503.02101/) (2015)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel](https://mlsanity.com/paper/1810.05369/) (2018)
- [On the linearity of large non-linear models: when and why the tangent kernel is constant](https://mlsanity.com/paper/2010.01092/) (2020)
- [Theoretical insights into the optimization landscape of over-parameterized shallow neural networks](https://mlsanity.com/paper/1707.04926/) (2017)
- [On the Power of Over-parametrization in Neural Networks with Quadratic Activation](https://mlsanity.com/paper/1803.01206/) (2018)
- [Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't](https://mlsanity.com/paper/2009.10713/) (2020)
- [What Can ResNet Learn Efficiently, Going Beyond Kernels?](https://mlsanity.com/paper/1905.10337/) (2019)
- [Learning One-hidden-layer Neural Networks with Landscape Design](https://mlsanity.com/paper/1711.00501/) (2017)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [Gradient Descent Can Take Exponential Time to Escape Saddle Points](https://mlsanity.com/paper/1705.10412/) (2017)
- [A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks](https://mlsanity.com/paper/2001.11443/) (2020)
- [A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth](https://mlsanity.com/paper/2003.05508/) (2020)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [Mean Field Analysis of Deep Neural Networks](https://mlsanity.com/paper/1903.04440/) (2019)
- [On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime](https://mlsanity.com/paper/2005.13530/) (2020)
- [Global optimality conditions for deep neural networks](https://mlsanity.com/paper/1707.02444/) (2017)
- [Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks](https://mlsanity.com/paper/1902.02880/) (2019)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks](https://mlsanity.com/paper/1910.02934/) (2019)
- [Convex Formulation of Overparameterized Deep Neural Networks](https://mlsanity.com/paper/1911.07626/) (2019)

## Built on by (most cited first, 12 of 12)

- [Neural Tangent Kernel Analysis of Deep Narrow Neural Networks](https://mlsanity.com/paper/2202.02981/) (2022)
- [Accelerating optimization over the space of probability measures](https://mlsanity.com/paper/2310.04006/) (2023)
- [Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport](https://mlsanity.com/paper/2403.12887/) (2024)
- [A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequality](https://mlsanity.com/paper/2311.15365/) (2023)
- [Global Convergence in Training Large-Scale Transformers](https://mlsanity.com/paper/2410.23610/) (2024)
- [Variational formulations of ODE-Net as a mean-field optimal control problem and existence results](https://mlsanity.com/paper/2303.05924/) (2023)
- [Correcting Auto-Differentiation in Neural-ODE Training](https://mlsanity.com/paper/2306.02192/) (2023)
- [Generalization of Scaled Deep ResNets in the Mean-Field Regime](https://mlsanity.com/paper/2403.09889/) (2024)
- [Training Infinitely Deep and Wide Transformers](https://mlsanity.com/paper/2605.17660/) (2026)
- [Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing](https://mlsanity.com/paper/2303.14244/) (2023)
- [Uniform Scaling Limits in AdamW-Trained Transformers](https://mlsanity.com/paper/2605.11059/) (2026)
- [Global Exponential Convergence of Two-Layer Linear Network Training](https://mlsanity.com/paper/2610.09356/) (2026)

## Abstract (arXiv)

Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first-order methods nevertheless find the global optimum in the overparameterized regime. We study this phenomenon with mean-field analysis, by translating the training process of ResNet to a gradient-flow partial differential equation (PDE) and examining the convergence properties of this limiting process. The activation function is assumed to be 2-homogeneous or partially 1-homogeneous; the regularized ReLU satisfies the latter condition. We show that if the ResNet is sufficiently large, with depth and width depending algebraically on the accuracy and confidence levels, first-order optimization methods can find global minimizers that fit the training data.