# Deep Residual Networks and Weight Initialization

arXiv: https://arxiv.org/abs/1709.02956 · HTML: https://mlsanity.com/paper/1709.02956/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2017.

## Built on by (most cited first, 12 of 12)

- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit](https://mlsanity.com/paper/2309.16620/) (2023)
- [Fast Certified Robust Training with Short Warmup](https://mlsanity.com/paper/2103.17268/) (2021)
- [Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning](https://mlsanity.com/paper/2106.05956/) (2021)
- [A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks](https://mlsanity.com/paper/2205.15307/) (2022)
- [AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks](https://mlsanity.com/paper/2109.08958/) (2021)
- [Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes](https://mlsanity.com/paper/2410.05052/) (2024)
- [Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks](https://mlsanity.com/paper/2602.07494/) (2026)
- [Training a universal instance segmentation network for live cell images of various cell types and imaging modalities](https://mlsanity.com/paper/2207.14347/) (2022)
- [Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initialization](https://mlsanity.com/paper/2304.03374/) (2023)
- [Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets](https://mlsanity.com/paper/2510.04327/) (2025)
- [LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks](https://mlsanity.com/paper/2607.14233/) (2026)

## Abstract (arXiv)

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.