mlsanityData to 7 Oct 2026

Efficient Transformers: A Survey

arXiv 2009.06732 ↗First posted 14 Sep 2020cs.LG cs.AI cs.CLcited by 784

Not on the map. Why?

Cited by 25 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of "X-former" models have been proposed - Reformer, Linformer, Performer, Longformer, to name a few - which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models, providing an organized and comprehensive overview of existing work and models across multiple domains.

Key facts. Cited by 784 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 146; 2022: 173; 2023: 164; 2024: 122; 2025: 102; 2026: 77.

784 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2021Transformers in Vision: A Surveycited by 1,009
  9. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  10. 2024VMamba: Visual State Space Modelcited by 955
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
38 more
  1. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  2. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2022Transformers in Time Series: A Surveycited by 489
  7. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  12. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  13. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  14. 2021Random Feature Attentioncited by 350
  15. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  16. 2023Adapting Language Models to Compress Contextscited by 350
  17. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  18. 2021Efficient Attentions for Long Document Summarizationcited by 342
  19. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  20. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  21. 2022Confident Adaptive Language Modelingcited by 297
  22. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  23. 2022Long Range Language Modeling via Gated State Spacescited by 265
  24. 2021Mesh Graphormercited by 264
  25. 2023Generating Images with Multimodal Language Modelscited by 259
  26. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  27. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  28. 2022Transformers in Medical Imaging: A Surveycited by 245
  29. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  30. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  31. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  32. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  33. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  34. 2022Memorizing Transformerscited by 219
  35. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  36. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  37. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  38. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202

50 most cited of 784 All 784 (JSON)