Deep Residual Networks and Weight Initialization
Not on the map. Why?
Abstract (arXiv)
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.
Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Sep 2017.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 2; 2023: 2; 2024: 1; 2025: 1; 2026: 2.
12 papers.
- 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
- 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
- 2021Fast Certified Robust Training with Short Warmupcited by 46
- 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
- 2022A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networkscited by 8
- 2021AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networkscited by 5
- 2024Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikescited by 4
- 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networkscited by 1
- 2022Training a universal instance segmentation network for live cell images of various cell types and imaging modalitiescited by 0
- 2023Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initializationcited by 0
- 2025Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNetscited by 0
- 2026LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networkscited by 0
- 2026LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks15 Jul 2026
- 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks7 Feb 2026
- 2025Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets5 Oct 2025
- 2024Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes7 Oct 2024
- 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit28 Sep 2023
- 2023Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initialization6 Apr 2023
- 2022Training a universal instance segmentation network for live cell images of various cell types and imaging modalities28 Jul 2022
- 2022A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks28 May 2022
- 2021AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks18 Sep 2021
- 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning10 Jun 2021
- 2021Fast Certified Robust Training with Short Warmup31 Mar 2021
- 2021Characterizing signal propagation to close the performance gap in unnormalized ResNets21 Jan 2021