mlsanityData to 7 Oct 2026

Generating Long Sequences with Sparse Transformers

arXiv 1904.10509 ↗First posted 23 Apr 2019cs.LG stat.MLcited by 1,523

Not on the map. Why?

Cited by 119 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to O(n √n). We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.

Key facts. Cited by 1,523 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 216; 2022: 203; 2023: 227; 2024: 311; 2025: 297; 2026: 269.

1,523 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023Mistral 7Bcited by 3,065
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  13. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  14. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  15. 2021Variational Diffusion Modelscited by 1,145
  16. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  17. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  18. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  19. 2021Transformers in Vision: A Surveycited by 1,009
  20. 2021Multiscale Vision Transformerscited by 896
  21. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  22. 2023Large Language Modelscited by 774
  23. 2021Going deeper with Image Transformerscited by 759
  24. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  25. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  26. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  27. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  28. 2023A Comprehensive Overview of Large Language Modelscited by 639
  29. 2021Score-based Generative Modeling in Latent Spacecited by 634
  30. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  31. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  32. 2022Efficiently Scaling Transformer Inferencecited by 529
  33. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  34. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  35. 2024Long Context Transfer from Language to Visioncited by 502
  36. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  37. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  38. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468

50 most cited of 1,523 All 1,523 (JSON)