# FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

arXiv: https://arxiv.org/abs/2603.05451 · HTML: https://mlsanity.com/paper/2603.05451/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://mlsanity.com/paper/2410.02367/) (2024)
- [SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization](https://mlsanity.com/paper/2411.10958/) (2024)
- [Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis](https://mlsanity.com/paper/2501.12084/) (2025)

## Built on by (most cited first, 30 of 66)

- [AVO: Agentic Variation Operators for Autonomous Evolutionary Search](https://mlsanity.com/paper/2603.24517/) (2026)
- [Bernini: Latent Semantic Planning for Video Diffusion](https://mlsanity.com/paper/2605.22344/) (2026)
- [LPM 1.0: Video-based Character Performance Model](https://mlsanity.com/paper/2604.07823/) (2026)
- [PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations](https://mlsanity.com/paper/2604.27472/) (2026)
- [OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization](https://mlsanity.com/paper/2605.17757/) (2026)
- [Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification](https://mlsanity.com/paper/2607.24027/) (2026)
- [M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling](https://mlsanity.com/paper/2603.14360/) (2026)
- [Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention](https://mlsanity.com/paper/2607.04422/) (2026)
- [CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution](https://mlsanity.com/paper/2608.12629/) (2026)
- [Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion](https://mlsanity.com/paper/2605.12825/) (2026)
- [ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention](https://mlsanity.com/paper/2605.23081/) (2026)
- [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](https://mlsanity.com/paper/2605.17170/) (2026)
- [Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing](https://mlsanity.com/paper/2607.19064/) (2026)
- [VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation](https://mlsanity.com/paper/2604.12798/) (2026)
- [FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training](https://mlsanity.com/paper/2606.22932/) (2026)
- [FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving](https://mlsanity.com/paper/2608.19758/) (2026)
- [Hardware-Aware FP4 FlashAttention-4](https://mlsanity.com/paper/2609.04105/) (2026)
- [VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention](https://mlsanity.com/paper/2609.15810/) (2026)
- [Rank-Factorized Implicit Neural Bias: Scaling Super-Resolution Transformer with FlashAttention](https://mlsanity.com/paper/2603.06738/) (2026)
- [Nautilus: An Auto-Scheduling Tensor Compiler for Efficient Tiled GPU Kernels](https://mlsanity.com/paper/2604.14825/) (2026)
- [Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU](https://mlsanity.com/paper/2604.15464/) (2026)
- [CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs](https://mlsanity.com/paper/2605.08467/) (2026)
- [Remember to Forget: Gated Adaptive Positional Encoding](https://mlsanity.com/paper/2605.10414/) (2026)
- [Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents](https://mlsanity.com/paper/2606.06453/) (2026)
- [HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference](https://mlsanity.com/paper/2607.04302/) (2026)
- [CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention](https://mlsanity.com/paper/2607.25291/) (2026)
- [WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning](https://mlsanity.com/paper/2607.28418/) (2026)
- [ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference](https://mlsanity.com/paper/2608.02947/) (2026)
- [Almost Free State Prediction Separation](https://mlsanity.com/paper/2609.03807/) (2026)
- [Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning](https://mlsanity.com/paper/2604.07242/) (2026)

## Abstract (arXiv)

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. While FlashAttention-3 optimized attention for Hopper GPUs through asynchronous execution and warp specialization, it primarily targets the H100 architecture. The AI industry has rapidly transitioned to deploying Blackwell-based systems such as the B200 and GB200, which exhibit fundamentally different performance characteristics due to asymmetric hardware scaling: tensor core throughput doubles while other functional units (shared memory bandwidth, exponential units) scale more slowly or remain unchanged. We develop several techniques to address these shifting bottlenecks on Blackwell GPUs: (1) redesigned pipelines that exploit fully asynchronous MMA operations and larger tile sizes, (2) software-emulated exponential and conditional softmax rescaling that reduces non-matmul operations, and (3) leveraging tensor memory and the 2-CTA MMA mode to reduce shared memory traffic and atomic adds in the backward pass. We demonstrate that our method, FlashAttention-4, achieves up to 1.3× speedup over cuDNN 9.13 and 2.7× over Triton on B200 GPUs with BF16, reaching up to 1613 TFLOPs/s (71% utilization). Beyond algorithmic innovations, we implement FlashAttention-4 entirely in CuTe-DSL embedded in Python, achieving 20-30× faster compile times compared to traditional C++ template-based approaches while maintaining full expressivity.