mlsanityData to 14 Oct 2026

The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures

arXiv 2510.05364 ↗First posted 6 Oct 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Transformers have dominated sequence processing tasks for the past seven years -- most notably language modeling. However, the inherent quadratic complexity of their attention mechanism remains a significant bottleneck as context length increases. This paper surveys recent efforts to overcome this bottleneck, including advances in (sub-quadratic) attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze these approaches in terms of compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 85 of its 119 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

85 of 119 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  13. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  14. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  15. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  16. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  17. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  19. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  20. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  21. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  22. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  23. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  24. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  25. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  26. 2024Titans: Learning to Memorize at Test Timecited by 328
  27. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  28. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  29. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  30. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  31. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  32. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  33. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  34. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  35. 2019Adaptively Sparse Transformerscited by 155
  36. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  37. 2023Representational Strengths and Limitations of Transformerscited by 133
  38. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?cited by 1
  2. 2026On Subquadratic Architectures: From Applications to Principlescited by 1
  3. 2026Hardware-Software Co-Design of Scalable, Energy-Efficient Analog Recurrent Computationscited by 0
  4. 2026ChronoSSM: Training for Temporally Aware Representations in Autoregressive State Space Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers
  2. 2025SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers
  3. 2025Log-Linear Attention
  4. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  5. 2025Forgetting Transformer: Softmax Attention with a Forget Gate
  6. 2021Saturated Transformers are Constant-Depth Threshold Circuits
  7. 2026Revisiting the Shape Convention of Transformer Language Models
  8. 2024From Attention to Activation: Unravelling the Enigmas of Large Language Models