# Norm-Preservation: Why Residual Networks Can Become Extremely Deep?

arXiv: https://arxiv.org/abs/1805.07477 · HTML: https://mlsanity.com/paper/1805.07477/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2018.

## Built on by (most cited first, 12 of 12)

- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [Field theory for optimal signal propagation in ResNets](https://mlsanity.com/paper/2305.07715/) (2023)
- [Learning Temporal Resolution in Spectrogram for Audio Classification](https://mlsanity.com/paper/2210.01719/) (2022)
- [Auto-Compressing Networks](https://mlsanity.com/paper/2506.09714/) (2025)
- [Auto-tuning of Deep Neural Networks by Conflicting Layer Removal](https://mlsanity.com/paper/2103.04331/) (2021)
- [SPI-GAN: Towards Single-Pixel Imaging through Generative Adversarial Network](https://mlsanity.com/paper/2107.01330/) (2021)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [A Lightweight and Gradient-Stable Neural Layer](https://mlsanity.com/paper/2106.04088/) (2021)
- [NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding](https://mlsanity.com/paper/2312.07507/) (2023)
- [Optical aberrations in autonomous driving: Physics-informed parameterized temperature scaling for neural network uncertainty calibration](https://mlsanity.com/paper/2412.13695/) (2024)
- [Contact Wasserstein Geodesics for Non-Conservative Schr"odinger Bridges](https://mlsanity.com/paper/2511.06856/) (2025)
- [Detecting Adversarial Data via Provable Adversarial Noise Amplification](https://mlsanity.com/paper/2605.02109/) (2026)

## Abstract (arXiv)

Augmenting neural networks with skip connections, as introduced in the so-called ResNet architecture, surprised the community by enabling the training of networks of more than 1,000 layers with significant performance gains. This paper deciphers ResNet by analyzing the effect of skip connections, and puts forward new theoretical results on the advantages of identity skip connections in neural networks. We prove that the skip connections in the residual blocks facilitate preserving the norm of the gradient, and lead to stable back-propagation, which is desirable from optimization perspective. We also show that, perhaps surprisingly, as more residual blocks are stacked, the norm-preservation of the network is enhanced. Our theoretical arguments are supported by extensive empirical evidence. Can we push for extra norm-preservation? We answer this question by proposing an efficient method to regularize the singular values of the convolution operator and making the ResNet's transition layers extra norm-preserving. Our numerical investigations demonstrate that the learning dynamics and the classification performance of ResNet can be improved by making it even more norm preserving. Our results and the introduced modification for ResNet, referred to as Procrustes ResNets, can be used as a guide for training deeper networks and can also inspire new deeper architectures.