mlsanityData to 14 Oct 2026

SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization

arXiv 2411.10958 ↗First posted 17 Nov 2024cs.LG cs.AI cs.CVcited by 114

Not on the map. Why?

Cited by 55 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Although quantization for linear layers has been widely used, its application to accelerate the attention process remains limited. To further enhance the efficiency of attention computation compared to SageAttention while maintaining precision, we propose SageAttention2, which utilizes significantly faster 4-bit matrix multiplication (Matmul) alongside additional precision-enhancing techniques. First, we propose to quantize matrices (Q, K) to INT4 in a hardware-friendly thread-level granularity and quantize matrices (widetilde P, V) to FP8. Second, we propose a method to smooth Q, enhancing the accuracy of INT4 QK^top. Third, we propose a two-level accumulation strategy for widetilde PV to enhance the accuracy of FP8 widetilde PV. The operations per second (OPS) of SageAttention2 surpass FlashAttention2 and xformers by about 3x and 4.5x on RTX4090, respectively. Moreover, SageAttention2 matches the speed of FlashAttention3(fp8) on the Hopper GPUs, while delivering much higher accuracy. Comprehensive experiments confirm that our approach incurs negligible end-to-end metrics loss across diverse models, including those for language, image, and video generation. The code is available at https://github.com/thu-ml/SageAttention.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Builds on 44 of its 72 references that are in the map. First posted on arXiv 17 Nov 2024.

Builds on

44 of 72 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2016Improved Techniques for Training GANscited by 2,970
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
32 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2020Rethinking Attention with Performerscited by 1,601
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  13. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  14. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  15. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  16. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  17. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  18. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  19. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  20. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  21. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  22. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  23. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 184
  24. 2018Online normalizer calculation for softmaxcited by 162
  25. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  26. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  27. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  28. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  29. 2024Diffusion Bridge Implicit Modelscited by 49
  30. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 28
  31. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 24
  32. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 49; 2026: 64.

114 papers.

Sort:

By area:

  1. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  2. 2025HunyuanVideo 1.5 Technical Reportcited by 160
  3. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 143
  4. 2025Fast Video Generation with Sliding Tile Attentioncited by 128
  5. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  6. 2025Mixture of Contexts for Long Video Generationcited by 95
  7. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  8. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  9. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 66
  10. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  11. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 66
  12. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
38 more
  1. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 53
  2. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  3. 2026HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worldscited by 44
  4. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  5. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timescited by 33
  6. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 30
  7. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  8. 2024Rethinking Token-wise Feature Caching: Accelerating Diffusion Transformers with Dual Feature Cachingcited by 28
  9. 2025Unleashing Vecset Diffusion Model for Fast Shape Generationcited by 26
  10. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  11. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 23
  12. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
  13. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 22
  14. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 22
  15. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  16. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 20
  17. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  18. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  19. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  20. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 16
  21. 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPUcited by 16
  22. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 15
  23. 2026Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Modelscited by 15
  24. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  25. 2025Turbo2K: Towards Ultra-Efficient and High-Quality 2K Video Synthesiscited by 13
  26. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  27. 2025UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformerscited by 12
  28. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 12
  29. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12
  30. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 12
  31. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 11
  32. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  33. 2025SAGE: A Framework of Precise Retrieval for RAGcited by 8
  34. 2025DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacecited by 8
  35. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 8
  36. 2026SageBwd: A Trainable Low-bit Attentioncited by 8
  37. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 8
  38. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
  2. 2026QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
  3. 2026RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
  4. 2024TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  6. 2026AVQ-Attention: Adaptive Vector-Quantized Attention
  7. 2026HyQuant: Hybrid-Precision Quantization for LLM Attention
  8. 2025FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration