mlsanityData to 7 Oct 2026

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

arXiv 2307.08691 ↗First posted 17 Jul 2023cs.LGcited by 2,551

Not on the map. Why?

Cited by 421 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4× compared to optimized baselines), with no approximation. However, FlashAttention is still not nearly as fast as optimized matrix-multiply (GEMM) operations, reaching only 25-40% of the theoretical maximum FLOPs/s. We observe that the inefficiency is due to suboptimal work partitioning between different thread blocks and warps on the GPU, causing either low-occupancy or unnecessary shared memory reads/writes. We propose FlashAttention-2, with better work partitioning to address these issues. In particular, we (1) tweak the algorithm to reduce the number of non-matmul FLOPs (2) parallelize the attention computation, even for a single head, across different thread blocks to increase occupancy, and (3) within each thread block, distribute the work between warps to reduce communication through shared memory. These yield around 2× speedup compared to FlashAttention, reaching 50-73% of the theoretical maximum FLOPs/s on A100 and getting close to the efficiency of GEMM operations. We empirically validate that when used end-to-end to train GPT-style models, FlashAttention-2 reaches training speed of up to 225 TFLOPs/s per A100 GPU (72% model FLOPs utilization).

Key facts. Cited by 2,551 papers in the map (data through 7 Oct 2026). Builds on 17 of its 20 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

17 of 20 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
5 more
  1. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  2. 2018Online normalizer calculation for softmaxcited by 154
  3. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  4. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  5. 2018Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarkingcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 86; 2024: 647; 2025: 910; 2026: 908.

2,551 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2024VMamba: Visual State Space Modelcited by 955
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  9. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  10. 2024Chronos: Learning the Language of Time Seriescited by 712
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
38 more
  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  5. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  6. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  9. 2023The Falcon Series of Open Language Modelscited by 526
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2024Long Context Transfer from Language to Visioncited by 502
  13. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  15. 2025LIMO: Less is More for Reasoningcited by 488
  16. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  17. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  18. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  19. 2023Llemma: An Open Language Model For Mathematicscited by 385
  20. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  21. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  22. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  23. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  24. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 356
  25. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  26. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  27. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  28. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  29. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  30. 2024Titans: Learning to Memorize at Test Timecited by 303
  31. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  32. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  33. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  34. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  35. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  36. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  37. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  38. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272

50 most cited of 2,551 All 2,551 (JSON)