mlsanityData to 7 Oct 2026

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

arXiv 2006.16236 ↗First posted 29 Jun 2020cs.LG stat.MLcited by 1,979

Not on the map. Why?

Cited by 117 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from Oleft(N2right) to Oleft(Nright), where N is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Key facts. Cited by 1,979 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 138; 2022: 181; 2023: 255; 2024: 406; 2025: 455; 2026: 544.

1,979 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  7. 2024VMamba: Visual State Space Modelcited by 955
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  11. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
38 more
  1. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  2. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  3. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  5. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  10. 2021A Survey of Transformerscited by 399
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  13. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  14. 2021Random Feature Attentioncited by 350
  15. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  16. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  17. 2021Efficient Attentions for Long Document Summarizationcited by 342
  18. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  19. 2021Choose a Transformer: Fourier or Galerkincited by 333
  20. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  21. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  22. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  23. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  24. 2024Titans: Learning to Memorize at Test Timecited by 303
  25. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  26. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  27. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 299
  28. 2022Long Range Language Modeling via Gated State Spacescited by 265
  29. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  30. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 251
  31. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  32. 2022Transformers in Medical Imaging: A Surveycited by 245
  33. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  34. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  35. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  36. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  37. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  38. 2022cosFormer: Rethinking Softmax in Attentioncited by 212

50 most cited of 1,979 All 1,979 (JSON)