mlsanityData to 14 Oct 2026

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

arXiv 2206.03126 ↗First posted 7 Jun 2022cs.LGcited by 141

Not on the map. Why?

Abstract (arXiv)

Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers - the distinctive architectural component of Transformers - can result in rank collapse of the tokens' representations at initialization. The question of if and how rank collapse affects training is still largely unanswered, and its investigation is necessary for a more comprehensive understanding of this architecture. In this work, we shed new light on the causes and the effects of this phenomenon. First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization. Furthermore, we provide a thorough description of the origin of rank collapse and discuss how to prevent it via an appropriate depth-dependent scaling of the residual branches. Finally, our analysis unveils that specific architectural hyperparameters affect the gradients of queries and values differently, leading to disproportionate gradient norms. This suggests an explanation for the widespread use of adaptive methods for Transformers' optimization.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). Builds on 38 of its 60 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

38 of 60 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2020On Layer Normalization in the Transformer Architecturecited by 922
26 more
  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  2. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  5. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  6. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  8. 2018Music Transformercited by 211
  9. 2020Understanding the Difficulty of Training Transformerscited by 210
  10. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  11. 2016Deep Information Propagationcited by 195
  12. 2021Neural Symbolic Regression that Scalescited by 192
  13. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  14. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  15. 2022Formal Mathematics Statement Curriculum Learningcited by 142
  16. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  17. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  18. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 102
  19. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  20. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  21. 2020Stable ResNetcited by 49
  22. 2019How to Initialize your Network? Robust Initialization for WeightNorm & ResNetscited by 36
  23. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 29
  24. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  25. 2017Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variablescited by 20
  26. 2021Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 31; 2025: 43; 2026: 50.

141 papers.

Sort:

By area:

  1. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  2. 2025Why do LLMs attend to the first token?cited by 127
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  4. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  5. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  6. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  7. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  8. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  9. 2025The Curse of Depth in Large Language Modelscited by 62
  10. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  11. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  12. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
38 more
  1. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 44
  2. 2025In Search of Adam's Secret Saucecited by 40
  3. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  4. 2023Simplifying Transformer Blockscited by 37
  5. 2025The Mean-Field Dynamics of Transformerscited by 30
  6. 2024Measure-to-measure interpolation using Transformerscited by 29
  7. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 28
  8. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  9. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  10. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  11. 2023LEMON: Lossless model expansioncited by 23
  12. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  13. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  14. 2025Long-Context Generalization with Sparse Attentioncited by 21
  15. 2025Critical attention scaling in long-context transformerscited by 20
  16. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  17. 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
  18. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
  19. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  20. 2025HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalizationcited by 16
  21. 2025NoT: Federated Unlearning via Weight Negationcited by 15
  22. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  23. 2023Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problemscited by 14
  24. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  25. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  26. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 13
  27. 2023Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generationcited by 12
  28. 2024When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Modelscited by 12
  29. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  30. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  31. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  32. 2024Unraveling the Gradient Descent Dynamics of Transformerscited by 11
  33. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 11
  34. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 10
  35. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  36. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  37. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9
  38. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9

50 most cited of 141 All 141 (JSON)