mlsanityData to 7 Oct 2026

Rethinking Skip Connection with Layer Normalization in Transformers and ResNets

arXiv 2105.07205 ↗First posted 15 May 2021cs.LG cs.CL cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Skip connection, is a widely-used technique to improve the performance and the convergence of deep neural networks, which is believed to relieve the difficulty in optimization due to non-linearity by propagating a linear component through the neural network layers. However, from another point of view, it can also be seen as a modulating mechanism between the input and the output, with the input scaled by a pre-defined value one. In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could be addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection. Inspired by the findings, we further propose to adaptively adjust the scale of the input by recursively applying skip connection with layer normalization, which promotes the performance substantially and generalizes well across diverse tasks including both machine translation and image classification datasets.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 15 May 2021.

Builds on

22 of 28 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  10. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  11. 2015Highway Networkscited by 257
  12. 2019Understanding and Improving Layer Normalizationcited by 213
10 more
  1. 2020Understanding the Difficulty of Training Transformerscited by 210
  2. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  3. 2019The Evolved Transformercited by 180
  4. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  5. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  6. 2019Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representationscited by 36
  7. 2018Exploiting Deep Representations for Neural Machine Translationcited by 23
  8. 2018simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captionscited by 20
  9. 2019MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learningcited by 19
  10. 2020Exploring and Distilling Cross-Modal Information for Image Captioningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 5; 2024: 4; 2025: 4; 2026: 1.

22 papers.

Sort:

By area:

  1. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  2. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  3. 2021MISSFormer: An Effective Medical Image Segmentation Transformercited by 103
  4. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  5. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  6. 2023Quantifying Context Mixing in Transformerscited by 43
  7. 2023Variational Annealing on Graphs for Combinatorial Optimizationcited by 21
  8. 2022Prophet Attention: Predicting Attention with Future Attention for Image Captioningcited by 10
  9. 2022Graph-in-Graph Network for Automatic Gene Ontology Description Generationcited by 5
  10. 2025Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attentioncited by 4
  11. 2025Global Regulation and Excitation via Attention Tuning for Stereo Matchingcited by 4
  12. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
10 more
  1. 2024stEnTrans: Transformer-based deep learning for spatial transcriptomics enhancementcited by 3
  2. 2023Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labelscited by 2
  3. 2022Rethinking skip connection model as a learnable Markov chaincited by 1
  4. 2024Federated Learning with Flexible Architecturescited by 1
  5. 2023Subject-Independent Classification of Brain Signals using Skip Connectionscited by 0
  6. 2023Gaussian Image Anomaly Detection with Greedy Eigencomponent Selectioncited by 0
  7. 2024Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classificationcited by 0
  8. 2024Transformer-based Capacity Prediction for Lithium-ion Batteries with Data Augmentationcited by 0
  9. 2025SC-Net: Robust Correspondence Learning via Spatial and Cross-Channel Contextcited by 0
  10. 2026Geometry Reinforced Efficient Attention Tuning Equipped with Normals for Robust Stereo Matchingcited by 0