# Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

arXiv: https://arxiv.org/abs/2006.05939 · HTML: https://mlsanity.com/paper/2006.05939/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 4 of 4)

- [CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters](https://mlsanity.com/paper/2203.15331/) (2022)
- [TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretraining](https://mlsanity.com/paper/2110.13492/) (2021)
- [On the Provable Generalization of Recurrent Neural Networks](https://mlsanity.com/paper/2109.14142/) (2021)
- [Automated 3D Tumor Segmentation using Temporal Cubic PatchGAN (TCuP-GAN)](https://mlsanity.com/paper/2311.14148/) (2023)

## Abstract (arXiv)

The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to ``guarantee" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works citefreeman2016topology, it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number m of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very ``shallow". The ``depth" of these local minima are at most O(m(η-1)/n), where n is the input dimension, η<1. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.