mlsanityData to 7 Oct 2026

Linformer: Self-Attention with Linear Complexity

arXiv 2006.04768 ↗First posted 8 Jun 2020cs.LG stat.MLcited by 1,462

Not on the map. Why?

Cited by 59 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses O(n2) time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from O(n2) to O(n) in both time and space. The resulting linear transformer, the textitLinformer, performs on par with standard Transformer models, while being much more memory- and time-efficient.

Key facts. Cited by 1,462 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 190; 2022: 269; 2023: 252; 2024: 303; 2025: 270; 2026: 178.

1,462 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2021ViViT: A Video Vision Transformercited by 1,610
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
38 more
  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2024VMamba: Visual State Space Modelcited by 955
  4. 2021Multiscale Vision Transformerscited by 896
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  6. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  7. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  10. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  11. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  12. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  13. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  14. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  15. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  17. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  18. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  19. 2021OCR-free Document Understanding Transformercited by 400
  20. 2021A Survey of Transformerscited by 399
  21. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  22. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  23. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  24. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  25. 2021Random Feature Attentioncited by 350
  26. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  27. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  28. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  29. 2021Choose a Transformer: Fourier or Galerkincited by 333
  30. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  31. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 327
  32. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  33. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  34. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  35. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  36. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  37. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  38. 2022Long Range Language Modeling via Gated State Spacescited by 265

50 most cited of 1,462 All 1,462 (JSON)