mlsanityData to 7 Oct 2026

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

arXiv 2407.08608 ↗First posted 11 Jul 2024cs.LG cs.AIcited by 507

Not on the map. Why?

Cited by 85 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2 achieving only 35% utilization on the H100 GPU. We develop three main techniques to speed up attention on Hopper GPUs: exploiting asynchrony of the Tensor Cores and TMA to (1) overlap overall computation and data movement via warp-specialization and (2) interleave block-wise matmul and softmax operations, and (3) block quantization and incoherent processing that leverages hardware support for FP8 low-precision. We demonstrate that our method, FlashAttention-3, achieves speedup on H100 GPUs by 1.5-2.0× with FP16 reaching up to 740 TFLOPs/s (75% utilization), and with FP8 reaching close to 1.2 PFLOPs/s. We validate that FP8 FlashAttention-3 achieves 2.6× lower numerical error than a baseline FP8 attention.

Key facts. Cited by 507 papers in the map (data through 7 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

50 of 61 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
38 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2020Efficient Transformers: A Surveycited by 784
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  13. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  14. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  15. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  16. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  19. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  20. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  21. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  22. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  23. 2021Random Feature Attentioncited by 350
  24. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  25. 2024Massive Activations in Large Language Modelscited by 277
  26. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  27. 2022FP8 Formats for Deep Learningcited by 187
  28. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  29. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  30. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  31. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  32. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  33. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  34. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  35. 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 23
  36. 2023Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPUcited by 22
  37. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  38. 2024Is Flash Attention Stable?cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 204; 2026: 264.

507 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  3. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  4. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  5. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  6. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  7. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  8. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  9. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  10. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  11. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  12. 2025HunyuanVideo 1.5 Technical Reportcited by 148
38 more
  1. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 142
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  4. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  5. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  6. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 115
  7. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  8. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  9. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  10. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  11. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  12. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  13. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  14. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  15. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  16. 2025LongCat-Video Technical Reportcited by 73
  17. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  18. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  19. 2025Pretraining Large Language Models with NVFP4cited by 67
  20. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  21. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  22. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  23. 2025Kling-Omni Technical Reportcited by 60
  24. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  25. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 57
  26. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  27. 2025AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformerscited by 49
  28. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  29. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  30. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  31. 20254DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videoscited by 46
  32. 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 46
  33. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  34. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 42
  35. 2025CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimizationcited by 42
  36. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  37. 2025Log-Linear Attentioncited by 39
  38. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36

50 most cited of 507 All 507 (JSON)