mlsanityData to 14 Oct 2026

SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training

arXiv 2505.11594 ↗First posted 16 May 2025cs.LG cs.AI cs.ARcited by 66

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The efficiency of attention is important due to its quadratic time complexity. We enhance the efficiency of attention through two key contributions: First, we leverage the new FP4 Tensor Cores in Blackwell GPUs to accelerate attention computation. Our implementation achieves 1038 TOPS on RTX5090, which is a 5x speedup over the fastest FlashAttention on RTX5090. Experiments show that our FP4 attention can accelerate inference of various models in a plug-and-play way. Second, we pioneer low-bit attention to training tasks. Existing low-bit attention works like FlashAttention3 and SageAttention focus only on inference. However, the efficiency of training large models is also important. To explore whether low-bit attention can be effectively applied to training tasks, we design an accurate and efficient 8-bit attention for both forward and backward propagation. Experiments indicate that 8-bit attention achieves lossless performance in fine-tuning tasks but exhibits slower convergence in pretraining tasks. The code is available at https://github.com/thu-ml/SageAttention.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

45 of 65 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
33 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  10. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  11. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  13. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  14. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  15. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  16. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  17. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  18. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  19. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  20. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  21. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  22. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  23. 2022FP8 Formats for Deep Learningcited by 196
  24. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 184
  25. 2018Online normalizer calculation for softmaxcited by 162
  26. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  27. 2023Microscaling Data Formats for Deep Learningcited by 152
  28. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  29. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  30. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  31. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  32. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  33. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 49.

66 papers.

Sort:

By area:

  1. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  2. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  3. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timescited by 33
  4. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  5. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 29
  6. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 22
  7. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  8. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 19
  9. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  10. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  11. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  12. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 12
38 more
  1. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  2. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 8
  3. 2026M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantizationcited by 8
  4. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 8
  5. 2026SageBwd: A Trainable Low-bit Attentioncited by 8
  6. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 8
  7. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
  8. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  9. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsificationcited by 7
  10. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 7
  11. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  12. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  13. 2026Ring Forcing: Towards Precise Long-Term Memory for Autoregressive Video Diffusioncited by 5
  14. 2026Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generationcited by 5
  15. 2025FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Modelscited by 4
  16. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantizationcited by 4
  17. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  18. 2026DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Trainingcited by 4
  19. 2026ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attentioncited by 4
  20. 2026Vidu S1: A Real-Time Interactive Video Generation Modelcited by 4
  21. 2025IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inferencecited by 3
  22. 2025Transform Trained Transformer for Accelerating Native 4K Video Generationcited by 3
  23. 2025Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animationcited by 3
  24. 2026VMonarch: Efficient Video Diffusion Transformers with Structured Attentioncited by 3
  25. 2026Search Your Block Floating Point Scales!cited by 3
  26. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Trainingcited by 2
  27. 2026HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Accelerationcited by 2
  28. 2026Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4cited by 2
  29. 2026LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Modelscited by 2
  30. 2026Speculative Decoding for Autoregressive Video Generationcited by 2
  31. 2026MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attentioncited by 2
  32. 2026Hardware-Aware FP4 FlashAttention-4cited by 2
  33. 2026VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentioncited by 2
  34. 2025Tensor-Compressed and Fully-Quantized Training of Neural PDE Solverscited by 1
  35. 2026BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformerscited by 1
  36. 2026Diagonal-Tiled Mixed-Precision Attention for Efficient Low-Bit MXFP Inferencecited by 1
  37. 2026HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusioncited by 1
  38. 2026Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
  2. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  3. 2026HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
  4. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference
  5. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs
  6. 2025Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention
  7. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  8. 2026SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference