# SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

arXiv: https://arxiv.org/abs/2502.18137 · HTML: https://mlsanity.com/paper/2502.18137/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 62 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 30 of 43)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory](https://mlsanity.com/paper/2402.04617/) (2024)
- [Online normalizer calculation for softmax](https://mlsanity.com/paper/1805.02867/) (2018)
- [DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics](https://mlsanity.com/paper/2310.13268/) (2023)
- [Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models](https://mlsanity.com/paper/2405.04233/) (2024)

## Built on by (most cited first, 30 of 120)

- [InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation](https://mlsanity.com/paper/2509.24663/) (2025)
- [IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse](https://mlsanity.com/paper/2603.12201/) (2026)
- [Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers](https://mlsanity.com/paper/2509.07120/) (2025)
- [SageAttention2++: A More Efficient Implementation of SageAttention2](https://mlsanity.com/paper/2505.21136/) (2025)
- [SLA2: Sparse-Linear Attention with Learnable Routing and QAT](https://mlsanity.com/paper/2602.12675/) (2026)
- [Bidirectional Sparse Attention for Faster Video Diffusion Training](https://mlsanity.com/paper/2509.01085/) (2025)
- [SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning](https://mlsanity.com/paper/2602.13515/) (2026)
- [Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning](https://mlsanity.com/paper/2511.08577/) (2025)
- [AVGGT: Rethinking Global Attention for Accelerating VGGT](https://mlsanity.com/paper/2512.02541/) (2025)
- [S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation](https://mlsanity.com/paper/2508.04016/) (2025)
- [SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning](https://mlsanity.com/paper/2508.06447/) (2025)
- [BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation](https://mlsanity.com/paper/2508.10774/) (2025)
- [Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention](https://mlsanity.com/paper/2602.04789/) (2026)
- [Geometry-Aware Rotary Position Embedding for Consistent Video World Model](https://mlsanity.com/paper/2602.07854/) (2026)
- [Mode Seeking meets Mean Seeking for Fast Long Video Generation](https://mlsanity.com/paper/2602.24289/) (2026)
- [HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention](https://mlsanity.com/paper/2603.28458/) (2026)
- [LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation](https://mlsanity.com/paper/2510.08318/) (2025)
- [SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing](https://mlsanity.com/paper/2603.08982/) (2026)
- [Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification](https://mlsanity.com/paper/2607.24027/) (2026)
- [Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers](https://mlsanity.com/paper/2601.11641/) (2026)
- [SageBwd: A Trainable Low-bit Attention](https://mlsanity.com/paper/2603.02170/) (2026)
- [QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification](https://mlsanity.com/paper/2509.23681/) (2025)
- [SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space](https://mlsanity.com/paper/2511.20102/) (2025)
- [Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers](https://mlsanity.com/paper/2512.16615/) (2025)
- [Efficient Context Scaling with LongCat ZigZag Attention](https://mlsanity.com/paper/2512.23966/) (2025)
- [ZEUS: Accelerating Diffusion Models with Only Second-Order Predictor](https://mlsanity.com/paper/2604.01552/) (2026)
- [EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval](https://mlsanity.com/paper/2510.18546/) (2025)
- [Sparser Block-Sparse Attention via Token Permutation](https://mlsanity.com/paper/2510.21270/) (2025)
- [RainFusion2.0: Temporal-Spatial Awareness and Hardware-Efficient Block-wise Sparse Attention](https://mlsanity.com/paper/2512.24086/) (2025)
- [Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms](https://mlsanity.com/paper/2603.28489/) (2026)

## Abstract (arXiv)

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The code is available at https://github.com/thu-ml/SpargeAttn.