# Identity Mappings in Deep Residual Networks

arXiv: https://arxiv.org/abs/1603.05027 · HTML: https://mlsanity.com/paper/1603.05027/

## Key facts

- Cited by 1,965 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2016.

## Built on by (most cited first, 30 of 1965)

- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications](https://mlsanity.com/paper/2209.02976/) (2022)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)

## Abstract (arXiv)

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers