mlsanityData to 7 Oct 2026

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

arXiv 2205.14135 ↗First posted 27 May 2022cs.LGcited by 3,611

Not on the map. Why?

Cited by 573 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that a missing principle is making attention algorithms IO-aware -- accounting for reads and writes between levels of GPU memory. We propose FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. We analyze the IO complexity of FlashAttention, showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes. We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method. FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3× speedup on GPT-2 (seq. length 1K), and 2.4× speedup on long-range arena (seq. length 1K-4K). FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1% accuracy).

Key facts. Cited by 3,611 papers in the map (data through 7 Oct 2026). Builds on 65 of its 111 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

65 of 111 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  12. 2020Efficient Transformers: A Surveycited by 784
  13. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  14. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  15. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  16. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  17. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  18. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  19. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  20. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  21. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  22. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  23. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  24. 2019Neural Legal Judgment Prediction in Englishcited by 208
  25. 2022Transformer Quality in Linear Timecited by 189
  26. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  27. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  28. 2020The Hardware Lotterycited by 160
  29. 2018Online normalizer calculation for softmaxcited by 154
  30. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  31. 2019Adaptive Attention Span in Transformerscited by 148
  32. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  33. 2019MLPerf Training Benchmarkcited by 114
  34. 2021An Attention Free Transformercited by 111
  35. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  36. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  37. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  38. 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 101

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 30; 2023: 371; 2024: 884; 2025: 1090; 2026: 1236.

3,611 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023Mistral 7Bcited by 3,065
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2023A Survey of Large Language Modelscited by 2,537
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  13. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  14. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  15. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  16. 2022Token Merging: Your ViT But Fastercited by 995
  17. 2024VMamba: Visual State Space Modelcited by 955
  18. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  19. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  20. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  21. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  22. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  23. 2023StarCoder: may the source be with you!cited by 790
  24. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  25. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  26. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  27. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  28. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  29. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  30. 2024Yi: Open Foundation Models by 01.AIcited by 706
  31. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  32. 2023A Comprehensive Overview of Large Language Modelscited by 639
  33. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  34. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  35. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  36. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  37. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  38. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598

50 most cited of 3,611 All 3,611 (JSON)