mlsanityData to 14 Oct 2026

Why Softmax Attention Outperforms Linear Attention

arXiv 2310.11685 ↗First posted 18 Oct 2023cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks. Among the pivotal components of the transformer architecture, the attention mechanism plays a crucial role in capturing token interactions within sequences through the utilization of softmax function. Conversely, linear attention presents a more computationally efficient alternative by approximating the softmax operation with linear complexity. However, it exhibits substantial performance degradation when compared to the traditional softmax attention mechanism. In this paper, we bridge the gap in our theoretical understanding of the reasons behind the practical performance gap between softmax and linear attention. By conducting a comprehensive comparative analysis of these two attention mechanisms, we shed light on the underlying reasons for why softmax attention outperforms linear attention in most scenarios.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 86 of its 112 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

86 of 112 references.

References · 86

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2020Reformer: The Efficient Transformercited by 1,668
  4. 2020Rethinking Attention with Performerscited by 1,601
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  13. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  14. 2018Universal Transformerscited by 647
  15. 2019Designing and Interpreting Probes with Control Taskscited by 640
  16. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  17. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  18. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  19. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  20. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  21. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  22. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  23. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  24. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  25. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  26. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  27. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  28. 2023Representational Strengths and Limitations of Transformerscited by 133
  29. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  30. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  31. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  32. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  33. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  34. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  35. 2023Fast Attention Requires Bounded Entriescited by 102
  36. 2022The Devil in Linear Transformercited by 98
  37. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  38. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  2. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  3. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  4. 2024Nonparametric Modern Hopfield Modelscited by 16
  5. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  6. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  7. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  8. 2026MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Headcited by 7
  9. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  10. 2025On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspectivecited by 6
  11. 2025Transformers Provably Learn Sparse XOR with Polylogarithmic Parameterscited by 3
  12. 2025Self-Adjust Softmaxcited by 3
3 more
  1. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 3
  2. 2025Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attentioncited by 2
  3. 2025Attention Condensation via Sparsity Induced Regularized Trainingcited by 0