mlsanityData to 7 Oct 2026

SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

arXiv 2502.18137 ↗First posted 25 Feb 2025cs.LG cs.AI cs.CVcited by 113

Not on the map. Why?

Cited by 43 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The code is available at https://github.com/thu-ml/SpargeAttn.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 43 of its 62 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

43 of 62 references.

References · 43

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
31 more
  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  3. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  4. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  10. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  11. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  12. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  13. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  14. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  15. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  16. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  17. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 159
  18. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  19. 2018Online normalizer calculation for softmaxcited by 154
  20. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  21. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  22. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  23. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  24. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  25. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 109
  26. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 63
  27. 2024Diffusion Bridge Implicit Modelscited by 47
  28. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  29. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  30. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 27
  31. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 86.

113 papers.

Sort:

By area:

  1. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  2. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  3. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 22
  4. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  5. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  6. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  7. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  8. 2025AVGGT: Rethinking Global Attention for Accelerating VGGTcited by 15
  9. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  10. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  11. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 12
  12. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
38 more
  1. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  2. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12
  3. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  4. 2025LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generationcited by 9
  5. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 9
  6. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 9
  7. 2026Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformerscited by 8
  8. 2026SageBwd: A Trainable Low-bit Attentioncited by 7
  9. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformerscited by 6
  10. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  11. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 6
  12. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsificationcited by 5
  13. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  14. 2025Sparser Block-Sparse Attention via Token Permutationcited by 5
  15. 2025RainFusion2.0: Temporal-Spatial Awareness and Hardware-Efficient Block-wise Sparse Attentioncited by 5
  16. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  17. 2025SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Spacecited by 4
  18. 2025PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generationcited by 4
  19. 2025Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondencecited by 4
  20. 2026AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generationcited by 4
  21. 2026RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformerscited by 4
  22. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformerscited by 4
  23. 2026Vidu S1: A Real-Time Interactive Video Generation Modelcited by 4
  24. 2026Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generationcited by 4
  25. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  26. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  27. 2025Transform Trained Transformer for Accelerating Native 4K Video Generationcited by 3
  28. 2025Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animationcited by 3
  29. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  30. 2026VMonarch: Efficient Video Diffusion Transformers with Structured Attentioncited by 3
  31. 2026EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecastingcited by 3
  32. 2026FlashBlock: Attention Caching for Efficient Long-Context Block Diffusioncited by 3
  33. 2026Prism: Spectral-Aware Block-Sparse Attentioncited by 3
  34. 2026MACRO: Advancing Multi-Reference Image Generation with Structured Long-Context Datacited by 3
  35. 2026ZEUS: Accelerating Diffusion Models with Only Second-Order Predictorcited by 3
  36. 2026LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attentioncited by 3
  37. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  38. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Stepscited by 3

50 most cited of 113 All 113 (JSON)