mlsanityData to 7 Oct 2026

Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

arXiv 2502.01776 ↗First posted 3 Feb 2025cs.CV cs.LGcited by 204

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 53 of its 69 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

53 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model28 Nov
  2. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality25 Oct
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  4. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Models12 Jun
  5. 2024Looking Backward: Streaming Video-to-Video Translation with Feature Banks24 May
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul

Other references · 47

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2023Consistency Modelscited by 1,778
35 more
  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  7. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  8. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  11. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  13. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  14. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  15. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  16. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  17. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  18. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  19. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  20. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  21. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  22. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 209
  23. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  24. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  25. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  26. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  27. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  28. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  29. 2023StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generationcited by 56
  30. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  31. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  32. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
  33. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  34. 2024Explorative Inbetweening of Time and Spacecited by 21
  35. 2024Condition-Aware Neural Network for Controlled Image Generationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 81; 2026: 123.

204 papers.

Sort:

By area:

  1. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  2. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  3. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  4. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  5. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  6. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  7. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  8. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  9. 2025Mixture of Contexts for Long Video Generationcited by 90
  10. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  11. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  12. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
38 more
  1. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  2. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  3. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  4. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  5. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  6. 2025FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolutioncited by 40
  7. 2025Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Cachingcited by 39
  8. 2025MagCache: Fast Video Generation with Magnitude-Aware Cachecited by 36
  9. 2025VMoBA: Mixture-of-Block Attention for Video Diffusion Modelscited by 35
  10. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  11. 2025DiCache: Let Diffusion Model Determine Its Own Cachecited by 34
  12. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 33
  13. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 27
  14. 2026Transition Matching Distillation for Fast Video Generationcited by 27
  15. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  16. 2025Pack and Force Your Memory: Long-form and Consistent Video Generationcited by 26
  17. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  18. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 21
  19. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  20. 2025Dual-Expert Consistency Model for Efficient and High-Quality Video Generationcited by 21
  21. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  22. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 20
  23. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 19
  24. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  25. 2026Flow caching for autoregressive video generationcited by 19
  26. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  27. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  28. 2025Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Lightcited by 16
  29. 2025OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Modelscited by 16
  30. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 16
  31. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 16
  32. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  33. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  34. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 15
  35. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 15
  36. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 15
  37. 2025DVD-Quant: Data-free Video Diffusion Transformers Quantizationcited by 14
  38. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 14

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
  2. 2026Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering
  3. 2026MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
  4. 2026ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
  5. 2025USV: Unified Sparsification for Accelerating Video Diffusion Models
  6. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
  7. 2026FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation
  8. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer