# Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

arXiv: https://arxiv.org/abs/1904.05263 · HTML: https://mlsanity.com/paper/1904.05263/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2019.

## Built on by (most cited first, 7 of 7)

- [Continuous vs. Discrete Optimization of Deep Neural Networks](https://mlsanity.com/paper/2107.06608/) (2021)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [What training reveals about neural network complexity](https://mlsanity.com/paper/2106.04186/) (2021)
- [Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks](https://mlsanity.com/paper/2206.02139/) (2022)
- [Image-to-Height Domain Translation for Synthetic Aperture Sonar](https://mlsanity.com/paper/2112.06307/) (2021)
- [Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks](https://mlsanity.com/paper/2204.07261/) (2022)
- [Asymptotic Analysis of Deep Residual Networks](https://mlsanity.com/paper/2212.08199/) (2022)

## Abstract (arXiv)

The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path are also established. As a consequence, it is shown that when the target function is in the reproducing kernel Hilbert space (RKHS) with a kernel defined by the initialization, there exist generalizable early-stopping solutions along the GD path. In addition, it is also shown that the GD path is uniformly close to the functions given by the related random feature model. Consequently, in this "implicit regularization" setting, the deep neural network model deteriorates to a random feature model. Our results hold for neural networks of any width larger than the input dimension.