mlsanityData to 7 Oct 2026

Fast Transformers with Clustered Attention

arXiv 2007.04825 ↗First posted 9 Jul 2020cs.LG stat.MLcited by 123

Not on the map. Why?

Abstract (arXiv)

Transformers have been proven a successful model for a variety of tasks in sequence modeling. However, computing the attention matrix, which is their key component, has quadratic complexity with respect to the sequence length, thus making them prohibitively expensive for large sequences. To address this, we propose clustered attention, which instead of computing the attention for every query, groups queries into clusters and computes attention just for the centroids. To further improve this approximation, we use the computed clusters to identify the keys with the highest attention per query and compute the exact key/query dot products. This results in a model with linear complexity with respect to the sequence length for a fixed number of clusters. We evaluate our approach on two automatic speech recognition datasets and show that our model consistently outperforms vanilla transformers for a given computational budget. Finally, we demonstrate that our model can approximate arbitrarily complex attention distributions with a minimal number of clusters by approximating a pretrained BERT model on GLUE and SQuAD benchmarks with only 25 clusters and no loss in performance.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 24; 2023: 24; 2024: 21; 2025: 10; 2026: 14.

123 papers.

Sort:

By area:

  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  3. 2021A Survey of Transformerscited by 399
  4. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  5. 2022Long Range Language Modeling via Gated State Spacescited by 265
  6. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  9. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 155
  10. 2021Learned Token Pruning for Transformerscited by 138
  11. 2023The emergence of clusters in self-attention dynamicscited by 133
  12. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
38 more
  1. 2022Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Cloudscited by 102
  2. 2021Finetuning Pretrained Transformers into RNNscited by 94
  3. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  4. 2022Flowformer: Linearizing Transformers with Conservation Flowscited by 69
  5. 2021Token Pooling in Vision Transformerscited by 68
  6. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  7. 2021Linear algebra with transformerscited by 62
  8. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  9. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesiscited by 55
  10. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  11. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  12. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  13. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  14. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
  15. 2021On the Expressive Power of Self-Attention Matricescited by 44
  16. 2021Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentationcited by 43
  17. 2023EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimationcited by 42
  18. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  19. 2024Scene Adaptive Sparse Transformer for Event-based Object Detectioncited by 39
  20. 2021Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequencescited by 38
  21. 2021Container: Context Aggregation Networkcited by 37
  22. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  23. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  24. 2021ABC: Attention with Bounded-memory Controlcited by 36
  25. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  26. 2024ThunderKittens: Simple, Fast, and Adorable AI Kernelscited by 34
  27. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  28. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  29. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  30. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  31. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  32. 2023Efficient Attention via Control Variatescited by 23
  33. 2021Transformer Uncertainty Estimation with Hierarchical Stochastic Attentioncited by 21
  34. 2022EcoFormer: Energy-Saving Attention with Linear Complexitycited by 21
  35. 2021RomeBERT: Robust Training of Multi-Exit BERTcited by 19
  36. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  37. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformerscited by 19
  38. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18

50 most cited of 123 All 123 (JSON)