Convergence Analysis of Deep Residual Networks
Not on the map. Why?
Abstract (arXiv)
Various powerful deep neural network architectures have made great contribution to the exciting successes of deep learning in the past two decades. Among them, deep Residual Networks (ResNets) are of particular importance because they demonstrated great usefulness in computer vision by winning the first place in many deep learning competitions. Also, ResNets were the first class of neural networks in the development history of deep learning that are really deep. It is of mathematical interest and practical meaning to understand the convergence of deep ResNets. We aim at characterizing the convergence of deep ResNets as the depth tends to infinity in terms of the parameters of the networks. Toward this purpose, we first give a matrix-vector description of general deep neural networks with shortcut connections and formulate an explicit expression for the networks by using the notions of activation domains and activation matrices. The convergence is then reduced to the convergence of two series involving infinite products of non-square matrices. By studying the two series, we establish a sufficient condition for pointwise convergence of ResNets. Our result is able to give justification for the design of ResNets. We also conduct experiments on benchmark machine learning data to verify our results.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 13 May 2022.
Builds on
26 of 42 references.
References · 26
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2014Going Deeper with Convolutionscited by 4,231
- 2016Identity Mappings in Deep Residual Networkscited by 1,960
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2018Universality of Deep Convolutional Neural Networkscited by 176
- 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
- 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
- 2016Identity Matters in Deep Learningcited by 107
- 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
- 2018Data Driven Governing Equations Approximation Using Deep Neural Networkscited by 94
- 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
14 more
- 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 78
- 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
- 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
- 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
- 2020The gap between theory and practice in function approximation with deep neural networkscited by 53
- 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
- 2019Nonlinear Approximation and (Deep) ReLU Networkscited by 45
- 2019Why ResNet Works? Residuals Generalizecited by 39
- 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
- 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
- 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
- 2021Convergence of Deep ReLU Networkscited by 14
- 2021Convergence of Deep Convolutional Neural Networkscited by 13
- 2020A Mathematical Introduction to Generative Adversarial Nets (GAN)cited by 8
Built on by
Citing papers per year: 2023: 1.
1 paper.