Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks
Not on the map. Why?
Abstract (arXiv)
We prove linear convergence of gradient descent to a global optimum for the training of deep residual networks with constant layer width and smooth activation function. We show that if the trained weights, as a function of the layer index, admit a scaling limit as the depth increases, then the limit has finite p-variation with p=2. Proofs are based on non-asymptotic estimates for the loss function and for norms of the network weights along the gradient descent path. We illustrate the relevance of our theoretical results to practical settings using detailed numerical experiments on supervised learning problems.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 14 Apr 2022.
Builds on
29 of 32 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2017Stable Architectures for Deep Neural Networkscited by 340
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
- 2016Deep Learning without Poor Local Minimacited by 233
- 2015The Power of Depth for Feedforward Neural Networkscited by 181
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
17 more
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
- 2016Identity Matters in Deep Learningcited by 107
- 2015Representation Benefits of Deep Feedforward Networkscited by 84
- 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
- 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
- 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
- 2020The Global Landscape of Neural Networks: An Overviewcited by 57
- 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
- 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
- 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
- 2017Depth Creates No Bad Local Minimacited by 29
- 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
- 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
- 2021Scaling Properties of Deep Residual Networkscited by 16
- 2018Representing smooth functions as compositions of near-identity functions with implications for deep network optimizationcited by 15
- 2018Non-attracting Regions of Local Minima in Deep and Wide Neural Networkscited by 8
- 2019Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connectionscited by 7
Built on by
Citing papers per year: 2022: 1; 2023: 1; 2024: 1.
3 papers.