mlsanityData to 7 Oct 2026

SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration

arXiv 2410.02367 ↗First posted 3 Oct 2024cs.LGcited by 158

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The transformer architecture predominates across various models. As the heart of the transformer, attention has a computational complexity of O(N2), compared to O(N) for linear transformations. When handling large sequence lengths, attention becomes the primary time-consuming component. Although quantization has proven to be an effective method for accelerating model inference, existing quantization methods primarily focus on optimizing the linear layer. In response, we first analyze the feasibility of quantization in attention detailedly. Following that, we propose SageAttention, a highly efficient and accurate quantization method for attention. The OPS (operations per second) of our approach outperforms FlashAttention2 and xformers by about 2.1 times and 2.7 times, respectively. SageAttention also achieves superior accuracy performance over FlashAttention3. Comprehensive experiments confirm that our approach incurs almost no end-to-end metrics loss across diverse models, including those for large language processing, image generation, and video generation. The codes are available at https://github.com/thu-ml/SageAttention.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 80 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

52 of 80 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  13. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  14. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  15. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  16. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  17. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  18. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  19. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  20. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  21. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  22. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  23. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  24. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  25. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  26. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  27. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  28. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  29. 2018Online normalizer calculation for softmaxcited by 154
  30. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  31. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  32. 2024UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peakscited by 64
  33. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  34. 2024Diffusion Bridge Implicit Modelscited by 47
  35. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantizationcited by 42
  36. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  37. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  38. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 67; 2026: 85.

158 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  3. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  4. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  5. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  6. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  7. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  8. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  9. 2025Mixture of Contexts for Long Video Generationcited by 90
  10. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  11. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  12. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
38 more
  1. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 63
  2. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  3. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  4. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  5. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  6. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  7. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  8. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  9. 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 51
  10. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  11. 2025Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Cachingcited by 39
  12. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 33
  13. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 33
  14. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  15. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timescited by 31
  16. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 27
  17. 2024Rethinking Token-wise Feature Caching: Accelerating Diffusion Transformers with Dual Feature Cachingcited by 26
  18. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  19. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 25
  20. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  21. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 22
  22. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  23. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 21
  24. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  25. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  26. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 20
  27. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19
  28. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 19
  29. 2025A Survey on Efficient Vision-Language Modelscited by 19
  30. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  31. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 17
  32. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  33. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  34. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  35. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 15
  36. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  37. 2025DVD-Quant: Data-free Video Diffusion Transformers Quantizationcited by 14
  38. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
  3. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
  4. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
  5. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  6. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  7. 2026KV Cache Compression Through the Lens of Transform Coding
  8. 2026LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformers