# Rethinking Skip Connection with Layer Normalization in Transformers and ResNets

arXiv: https://arxiv.org/abs/2105.07205 · HTML: https://mlsanity.com/paper/2105.07205/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 15 May 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Tensor2Tensor for Neural Machine Translation](https://mlsanity.com/paper/1803.07416/) (2018)
- [Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations](https://mlsanity.com/paper/1905.06139/) (2019)
- [Exploiting Deep Representations for Neural Machine Translation](https://mlsanity.com/paper/1810.10181/) (2018)
- [simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions](https://mlsanity.com/paper/1808.08732/) (2018)
- [MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning](https://mlsanity.com/paper/1911.09483/) (2019)
- [Exploring and Distilling Cross-Modal Information for Image Captioning](https://mlsanity.com/paper/2002.12585/) (2020)

## Built on by (most cited first, 22 of 22)

- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [VOLO: Vision Outlooker for Visual Recognition](https://mlsanity.com/paper/2106.13112/) (2021)
- [MISSFormer: An Effective Medical Image Segmentation Transformer](https://mlsanity.com/paper/2109.07162/) (2021)
- [Incorporating Residual and Normalization Layers into Analysis of Masked Language Models](https://mlsanity.com/paper/2109.07152/) (2021)
- [Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation](https://mlsanity.com/paper/2111.04318/) (2021)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [Variational Annealing on Graphs for Combinatorial Optimization](https://mlsanity.com/paper/2311.14156/) (2023)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [Graph-in-Graph Network for Automatic Gene Ontology Description Generation](https://mlsanity.com/paper/2206.05311/) (2022)
- [Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention](https://mlsanity.com/paper/2506.18335/) (2025)
- [Global Regulation and Excitation via Attention Tuning for Stereo Matching](https://mlsanity.com/paper/2509.15891/) (2025)
- [Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads](https://mlsanity.com/paper/2510.16807/) (2025)
- [stEnTrans: Transformer-based deep learning for spatial transcriptomics enhancement](https://mlsanity.com/paper/2407.08224/) (2024)
- [Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels](https://mlsanity.com/paper/2307.01969/) (2023)
- [Rethinking skip connection model as a learnable Markov chain](https://mlsanity.com/paper/2209.15278/) (2022)
- [Federated Learning with Flexible Architectures](https://mlsanity.com/paper/2406.09877/) (2024)
- [Subject-Independent Classification of Brain Signals using Skip Connections](https://mlsanity.com/paper/2301.07918/) (2023)
- [Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection](https://mlsanity.com/paper/2308.04944/) (2023)
- [Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification](https://mlsanity.com/paper/2404.00946/) (2024)
- [Transformer-based Capacity Prediction for Lithium-ion Batteries with Data Augmentation](https://mlsanity.com/paper/2407.16036/) (2024)
- [SC-Net: Robust Correspondence Learning via Spatial and Cross-Channel Context](https://mlsanity.com/paper/2512.23473/) (2025)
- [Geometry Reinforced Efficient Attention Tuning Equipped with Normals for Robust Stereo Matching](https://mlsanity.com/paper/2604.09142/) (2026)

## Abstract (arXiv)

Skip connection, is a widely-used technique to improve the performance and the convergence of deep neural networks, which is believed to relieve the difficulty in optimization due to non-linearity by propagating a linear component through the neural network layers. However, from another point of view, it can also be seen as a modulating mechanism between the input and the output, with the input scaled by a pre-defined value one. In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could be addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection. Inspired by the findings, we further propose to adaptively adjust the scale of the input by recursively applying skip connection with layer normalization, which promotes the performance substantially and generalizes well across diverse tasks including both machine translation and image classification datasets.