mlsanityData to 14 Oct 2026

VSA: Faster Video Diffusion with Trainable Sparse Attention

arXiv 2505.13389 ↗First posted 19 May 2025cs.CVcited by 104

On the map: Diffusion models → Video generation

Abstract (arXiv)

Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions. We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at both training and inference. In VSA, a lightweight coarse stage pools tokens into tiles and identifies high-weight critical tokens; a fine stage computes token-level attention only inside those tiles subjecting to block computing layout to ensure hard efficiency. This leads to a single differentiable kernel that trains end-to-end, requires no post-hoc profiling, and sustains 85% of FlashAttention3 MFU. We perform a large sweep of ablation studies and scaling-law experiments by pretraining DiTs from 60M to 1.4B parameters. VSA reaches a Pareto point that cuts training FLOPS by 2.53× with no drop in diffusion loss. Retrofitting the open-source Wan-2.1 model speeds up attention time by 6× and lowers end-to-end generation time from 31s to 18s with comparable quality. These results establish trainable sparse attention as a practical alternative to full attention and a key enabler for further scaling of video diffusion models. Code will be available at https://github.com/hao-ai-lab/FastVideo.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 46 of its 49 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

46 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generation28 Feb
  2. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile10 Feb
  3. 2025Goku: Flow Based Video Generative Foundation Models7 Feb
  4. 2025Fast Video Generation with Sliding Tile Attention6 Feb
  5. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity3 Feb
  6. 2025Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models14 Jan
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  8. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  9. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
25 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  9. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  11. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  12. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  13. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  14. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  15. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  16. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  17. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  18. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  19. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  20. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  21. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 151
  22. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  23. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  24. 2024ThunderKittens: Simple, Fast, and Adorable AI Kernelscited by 36
  25. 2025Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Lightcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 75.

104 papers.

Sort:

By area:

  1. 2025Training Agents Inside of Scalable World Modelscited by 136
  2. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 106
  3. 2025Mixture of Contexts for Long Video Generationcited by 95
  4. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 94
  5. 2025LongCat-Video Technical Reportcited by 85
  6. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  7. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 52
  8. 2025FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolutioncited by 43
  9. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  10. 2025Critique of World Modelcited by 25
  11. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 23
  12. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
38 more
  1. 2025Dual-Expert Consistency Model for Efficient and High-Quality Video Generationcited by 21
  2. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 19
  3. 2025OBS-Diff: Accurate Pruning For Diffusion Models in One-Shotcited by 19
  4. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 18
  5. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  6. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  7. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 17
  8. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 16
  9. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformercited by 15
  10. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  11. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 12
  12. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  13. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 12
  14. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12
  15. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  16. 2025LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generationcited by 11
  17. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 11
  18. 2026Speed3R: Sparse Feed-forward 3D Reconstruction Modelscited by 10
  19. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  20. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 9
  21. 2026Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillationcited by 8
  22. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 8
  23. 2026ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformerscited by 7
  24. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformercited by 7
  25. 2026Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generationcited by 7
  26. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 7
  27. 2025Block Cascading: Training Free Acceleration of Block-Causal Video Modelscited by 6
  28. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformerscited by 6
  29. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  30. 2026MonarchRT: Efficient Attention for Real-Time Video Generationcited by 6
  31. 2026AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generationcited by 6
  32. 2026EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editingcited by 5
  33. 2026HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generationcited by 5
  34. 2025FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Modelscited by 4
  35. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  36. 2025PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generationcited by 4
  37. 2026PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inferencecited by 4
  38. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
  2. 2026LVSA: Training-Free Sparse Attention for Long Video Diffusion
  3. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
  4. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation
  5. 2026Competition Domains Matter: Selective Head Globalization for Training-Free Sparse Attention in Video Diffusion Transformers
  6. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification
  7. 2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
  8. 2026MegaSlide-DiT: Memory-Centric Adaptation and Deformable Local Attention for Efficient Video Diffusion