mlsanityData to 14 Oct 2026

On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime

arXiv 2110.02926 ↗First posted 6 Oct 2021cs.LG cs.NA math.NAcited by 12

Not on the map. Why?

Abstract (arXiv)

Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first-order methods nevertheless find the global optimum in the overparameterized regime. We study this phenomenon with mean-field analysis, by translating the training process of ResNet to a gradient-flow partial differential equation (PDE) and examining the convergence properties of this limiting process. The activation function is assumed to be 2-homogeneous or partially 1-homogeneous; the regularized ReLU satisfies the latter condition. We show that if the ResNet is sufficiently large, with depth and width depending algebraically on the accuracy and confidence levels, first-order optimization methods can find global minimizers that fit the training data.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

32 of 43 references.

References · 32

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  8. 2017How to Escape Saddle Points Efficientlycited by 285
  9. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  10. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  11. 2016Deep Learning without Poor Local Minimacited by 234
  12. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
20 more
  1. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  2. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  3. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  4. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  5. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  6. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
  7. 2017The loss surface of deep and wide neural networkscited by 74
  8. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  9. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  10. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  11. 2019Machine Learning from a Continuous Viewpointcited by 50
  12. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  13. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
  14. 2017Global optimality conditions for deep neural networkscited by 31
  15. 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
  16. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  17. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
  18. 2019Convex Formulation of Overparameterized Deep Neural Networkscited by 16
  19. 2019Mean-Field Neural ODEs via Relaxed Optimal Controlcited by 16
  20. 2021When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 3; 2025: 0; 2026: 3.

12 papers.

  1. 2022Neural Tangent Kernel Analysis of Deep Narrow Neural Networkscited by 11
  2. 2023Accelerating optimization over the space of probability measurescited by 10
  3. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 8
  4. 2023A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequalitycited by 5
  5. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  6. 2023Variational formulations of ODE-Net as a mean-field optimal control problem and existence resultscited by 3
  7. 2023Correcting Auto-Differentiation in Neural-ODE Trainingcited by 3
  8. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  9. 2026Training Infinitely Deep and Wide Transformerscited by 1
  10. 2023Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensingcited by 0
  11. 2026Uniform Scaling Limits in AdamW-Trained Transformerscited by 0
  12. 2026Global Exponential Convergence of Two-Layer Linear Network Trainingcited by 0