mlsanityData to 14 Oct 2026

SageAttention2++: A More Efficient Implementation of SageAttention2

arXiv 2505.21136 ↗First posted 27 May 2025cs.LG cs.AI cs.ARcited by 22

Not on the map. Why?

Abstract (arXiv)

The efficiency of attention is critical because its time complexity grows quadratically with sequence length. SageAttention2 addresses this by utilizing quantization to accelerate matrix multiplications (Matmul) in attention. To further accelerate SageAttention2, we propose to utilize the faster instruction of FP8 Matmul accumulated in FP16. The instruction is 2x faster than the FP8 Matmul used in SageAttention2. Our experiments show that SageAttention2++ achieves a 3.9x speedup over FlashAttention while maintaining the same attention accuracy as SageAttention2. This means SageAttention2++ effectively accelerates various models, including those for language, image, and video generation, with negligible end-to-end metrics loss. The code will be available at https://github.com/thu-ml/SageAttention.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

38 of 46 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
26 more
  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  5. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  6. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  8. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  12. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  13. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  14. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  15. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 184
  16. 2018Online normalizer calculation for softmaxcited by 162
  17. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  18. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  19. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  20. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  21. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  22. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  23. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  24. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 24
  25. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  26. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 15.

22 papers.

Sort:

By area:

  1. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  2. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  3. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  4. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timescited by 33
  5. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  6. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  7. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 12
  8. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  9. 2026SageBwd: A Trainable Low-bit Attentioncited by 8
  10. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  11. 2026Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generationcited by 5
  12. 2025FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Modelscited by 4
10 more
  1. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  2. 2026Vidu S1: A Real-Time Interactive Video Generation Modelcited by 4
  3. 2026LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Modelscited by 2
  4. 2026Speculative Decoding for Autoregressive Video Generationcited by 2
  5. 2026MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attentioncited by 2
  6. 2026VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentioncited by 2
  7. 2026CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attentioncited by 1
  8. 2025A Lightweight Learned Cardinality Estimation Modelcited by 0
  9. 20266Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Modelscited by 0
  10. 2026Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
  2. 2026QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
  3. 2024TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
  4. 2026FlashAttention for Scalable Vector Architectures
  5. 2026RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
  6. 2026High-Rate Quantized Matrix Multiplication I
  7. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion
  8. 2026Vectorization Of Narrow Matrix Multiplication for Ascend AI Inference Acceleration