mlsanityData to 7 Oct 2026

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

arXiv 2103.03404 ↗First posted 5 Mar 2021cs.LGcited by 411

Not on the map. Why?

Abstract (arXiv)

Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures.

Key facts. Cited by 411 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 5 Mar 2021.

Builds on

36 of 56 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
24 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2015Pointer Networkscited by 882
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2018Universal Transformerscited by 613
  9. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  11. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  12. 2019Attention Augmented Convolutional Networkscited by 363
  13. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  14. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  15. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  16. 2020The Lipschitz Constant of Self-Attentioncited by 171
  17. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  18. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  19. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 56
  20. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  21. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  22. 2016Neural Text Generation from Structured Data with Application to the Biography Domaincited by 20
  23. 2020Simplified Self-Attention for Transformer-based End-to-End Speech Recognitioncited by 20
  24. 2018Lipschitz Networks and Distributional Robustnesscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 45; 2023: 63; 2024: 86; 2025: 73; 2026: 105.

411 papers.

Sort:

By area:

  1. 2022In-context Learning and Induction Headscited by 930
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  4. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  5. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  6. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  7. 2021A Survey of Transformerscited by 399
  8. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  9. 2022A Contrastive Framework for Neural Text Generationcited by 224
  10. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  11. 2021SpectralFormer: Rethinking Hyperspectral Image Classification with Transformerscited by 168
  12. 2023A mathematical perspective on Transformerscited by 160
38 more
  1. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  2. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  3. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  4. 2023TriDet: Temporal Action Detection with Relative Boundary Modelingcited by 141
  5. 2022On The Computational Complexity of Self-Attentioncited by 137
  6. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  7. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  8. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  9. 2023The emergence of clusters in self-attention dynamicscited by 133
  10. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  11. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  12. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  13. 2025Why do LLMs attend to the first token?cited by 122
  14. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 121
  15. 2021A Survey of Visual Transformerscited by 117
  16. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  17. 2022Block-Recurrent Transformerscited by 107
  18. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  19. 2021Unsupervised Discovery of Object Radiance Fieldscited by 100
  20. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  21. 2021P2T: Pyramid Pooling Transformer for Scene Understandingcited by 84
  22. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  23. 2022Unified Visual Transformer Compressioncited by 79
  24. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  25. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  26. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  27. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  28. 2023Transformers as Support Vector Machinescited by 67
  29. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  30. 2022Video Transformers: A Surveycited by 63
  31. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  32. 2024Dynamic Diffusion Transformercited by 60
  33. 2024LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioningcited by 58
  34. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  35. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  36. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  37. 2021Multi-view 3D Reconstruction with Transformercited by 52
  38. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 52

50 most cited of 411 All 411 (JSON)