mlsanityData to 7 Oct 2026

Efficient Content-Based Sparse Attention with Routing Transformers

arXiv 2003.05997 ↗First posted 12 Mar 2020cs.LG eess.AS stat.MLcited by 469

Not on the map. Why?

Cited by 24 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to Oleft(n1.5dright) from Oleft(n2dright) for sequence length n and hidden dimension d. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity) as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22 layer Routing Transformer model trained on sequences of length 8192.

Key facts. Cited by 469 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 90; 2022: 77; 2023: 78; 2024: 83; 2025: 75; 2026: 66.

469 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2021Variational Diffusion Modelscited by 1,145
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  10. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
38 more
  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  3. 2021A Survey of Transformerscited by 399
  4. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  7. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  8. 2021Random Feature Attentioncited by 350
  9. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  10. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  13. 2024Titans: Learning to Memorize at Test Timecited by 303
  14. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  15. 2022Long Range Language Modeling via Gated State Spacescited by 265
  16. 2021Generating Images with Sparse Representationscited by 262
  17. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  18. 2021Hash Layers For Large Sparse Modelscited by 236
  19. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  20. 2022Memorizing Transformerscited by 219
  21. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  22. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  23. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  24. 2023Learning A Sparse Transformer Network for Effective Image Derainingcited by 204
  25. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  26. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  27. 2022Transformer Quality in Linear Timecited by 189
  28. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  29. 2023Augmenting Language Models with Long-Term Memorycited by 186
  30. 2021Thinking Like Transformerscited by 182
  31. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  32. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  33. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  34. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  35. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  36. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  37. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  38. 2021A Dataset for Answering Time-Sensitive Questionscited by 150

50 most cited of 469 All 469 (JSON)