mlsanityData to 7 Oct 2026

Overparameterization of deep ResNet: zero loss and mean-field analysis

arXiv 2105.14417 ↗First posted 30 May 2021cs.LG cs.NA math.NAcited by 24

Not on the map. Why?

Abstract (arXiv)

Finding parameters in a deep neural network (NN) that fit training data is a nonconvex optimization problem, but a basic first-order optimization method (gradient descent) finds a global optimizer with perfect fit (zero-loss) in many practical situations. We examine this phenomenon for the case of Residual Neural Networks (ResNet) with smooth activation functions in a limiting regime in which both the number of layers (depth) and the number of weights in each layer (width) go to infinity. First, we use a mean-field-limit argument to prove that the gradient descent for parameter training becomes a gradient flow for a probability distribution that is characterized by a partial differential equation (PDE) in the large-NN limit. Next, we show that under certain assumptions, the solution to the PDE converges in the training time to a zero-loss solution. Together, these results suggest that the training of the ResNet gives a near-zero loss if the ResNet is large enough. We give estimates of the depth and width needed to reduce the loss below a given threshold, with high probability.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 30 May 2021.

Builds on

37 of 46 references.

References · 37

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2017How to Escape Saddle Points Efficientlycited by 284
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  10. 2016Deep Learning without Poor Local Minimacited by 233
  11. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  12. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
25 more
  1. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  2. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  3. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  4. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  5. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  6. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  7. 2017The loss surface of deep and wide neural networkscited by 74
  8. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  9. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  10. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  11. 2019Machine Learning from a Continuous Viewpointcited by 50
  12. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  13. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  14. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  15. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  16. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  17. 2017Global optimality conditions for deep neural networkscited by 31
  18. 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
  19. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
  20. 2019Convex Formulation of Overparameterized Deep Neural Networkscited by 16
  21. 2019Mean-Field Neural ODEs via Relaxed Optimal Controlcited by 16
  22. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  23. 2018Representing smooth functions as compositions of near-identity functions with implications for deep network optimizationcited by 15
  24. 2020When does gradient descent with logistic loss find interpolating two-layer networks?cited by 12
  25. 2021When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 10; 2024: 3; 2025: 3; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  2. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  3. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  4. 2022Neural Tangent Kernel Analysis of Deep Narrow Neural Networkscited by 11
  5. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
  6. 2021Stationary Density Estimation of Ito Diffusions Using Deep Learningcited by 9
  7. 2023Accelerating optimization over the space of probability measurescited by 9
  8. 2023Depth Separation with Multilayer Mean-Field Networkscited by 8
  9. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 7
  10. 2023High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariancecited by 5
  11. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  12. 2024Global Convergence in Training Large-Scale Transformerscited by 5
12 more
  1. 2023A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequalitycited by 4
  2. 2025A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Modelscited by 4
  3. 2022A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networkscited by 3
  4. 2023Variational formulations of ODE-Net as a mean-field optimal control problem and existence resultscited by 3
  5. 2023Boosting Residual Networks with Group Knowledgecited by 3
  6. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  7. 2023Correcting Auto-Differentiation in Neural-ODE Trainingcited by 1
  8. 2025Gradient Flow Convergence Guarantee for General Neural Network Architecturescited by 0
  9. 2026A Quantitative Approximation Framework for Flow Distillation in Diffusion Modelscited by 0
  10. 2026Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNetscited by 0
  11. 2026Nesterov acceleration in optimizing over probability measurescited by 0
  12. 2026Stochastic Gradient Descent over P2cited by 0