mlsanityData to 7 Oct 2026

Efficient Attention: Attention with Linear Complexities

arXiv 1812.01243 ↗First posted 4 Dec 2018cs.CV cs.AI cs.LGcited by 366

Not on the map. Why?

Cited by 33 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Dot-product attention has wide applications in computer vision and natural language processing. However, its memory and computational costs grow quadratically with the input size. Such growth prohibits its application on high-resolution inputs. To remedy this drawback, this paper proposes a novel efficient attention mechanism equivalent to dot-product attention but with substantially less memory and computational costs. Its resource efficiency allows more widespread and flexible integration of attention modules into a network, which leads to better accuracies. Empirical evaluations demonstrated the effectiveness of its advantages. Efficient attention modules brought significant performance boosts to object detectors and instance segmenters on MS-COCO 2017. Further, the resource efficiency democratizes attention to complex models, where high costs prohibit the use of dot-product attention. As an exemplar, a model with efficient attention achieved state-of-the-art accuracies for stereo depth estimation on the Scene Flow dataset. Code is available at https://github.com/cmsflash/efficient-attention.

Key facts. Cited by 366 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 34; 2022: 53; 2023: 55; 2024: 94; 2025: 80; 2026: 50.

366 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  4. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  5. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  6. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  7. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  8. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  9. 2021Choose a Transformer: Fourier or Galerkincited by 333
  10. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  11. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  12. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
38 more
  1. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  2. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  4. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  5. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  6. 2021Meta-learning via Language Model In-context Tuningcited by 163
  7. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  8. 2023Spike-driven Transformercited by 141
  9. 2022MatchFormer: Interleaving Attention in Transformers for Feature Matchingcited by 126
  10. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  11. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  12. 2022Transforming Model Prediction for Trackingcited by 117
  13. 2023FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editingcited by 105
  14. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  15. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 92
  16. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  17. 2023MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazingcited by 81
  18. 2023Diffusion Models Without Attentioncited by 75
  19. 2022HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imagingcited by 66
  20. 2023Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulationcited by 65
  21. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  22. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  23. 2022CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentationcited by 60
  24. 2023MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compressioncited by 56
  25. 2022Neural Texture Extraction and Distribution for Controllable Person Image Synthesiscited by 55
  26. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  27. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  28. 2023ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputationcited by 50
  29. 2024MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereocited by 49
  30. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  31. 2022Divert More Attention to Vision-Language Trackingcited by 47
  32. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  33. 2022EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Predictioncited by 42
  34. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  35. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  36. 2021Polarized Self-Attention: Towards High-quality Pixel-wise Regressioncited by 39
  37. 2024A Survey of Mambacited by 39
  38. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37

50 most cited of 366 All 366 (JSON)