mlsanityData to 14 Oct 2026

Fast Video Generation with Sliding Tile Attention

arXiv 2502.04507 ↗First posted 6 Feb 2025cs.CVcited by 128

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s without quality degradation, requiring no training. Enabling finetuning further lowers latency to 268s with only a 0.09% drop on VBench. We make our codebase public at https://github.com/hao-ai-lab/FastVideo.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 39 of its 50 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

39 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity3 Feb
  2. 2024CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up20 Dec
  3. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality25 Oct
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  5. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback29 May
  6. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  7. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  8. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep

Other references · 31

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  10. 2023Consistency Modelscited by 1,822
  11. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  12. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
19 more
  1. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  2. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  3. 2023Adversarial Diffusion Distillationcited by 759
  4. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  5. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  8. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 459
  9. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  10. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 219
  11. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  12. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 175
  13. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  14. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  15. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  16. 2024Multistep Distillation of Diffusion Models via Moment Matchingcited by 82
  17. 2024On the Content Bias in Fr'echet Video Distancecited by 52
  18. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 29
  19. 2021Learned Queries for Efficient Local Attentioncited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 55; 2026: 73.

128 papers.

Sort:

By area:

  1. 2025HunyuanVideo 1.5 Technical Reportcited by 160
  2. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  3. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  4. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 106
  5. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 104
  6. 2025Mixture of Contexts for Long Video Generationcited by 95
  7. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 94
  8. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  9. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  10. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  11. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  12. 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 54
38 more
  1. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  2. 2025VMoBA: Mixture-of-Block Attention for Video Diffusion Modelscited by 37
  3. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  4. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 34
  5. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  6. 2025Critique of World Modelcited by 25
  7. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  8. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
  9. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  10. 2025Dual-Expert Consistency Model for Efficient and High-Quality Video Generationcited by 21
  11. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  12. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 20
  13. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 20
  14. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  15. 2025Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Lightcited by 18
  16. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 18
  17. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  18. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  19. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 17
  20. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 17
  21. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 16
  22. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  23. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformercited by 15
  24. 2026Pathwise Test-Time Correction for Autoregressive Long Video Generationcited by 15
  25. 2025Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformerscited by 13
  26. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 13
  27. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  28. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 12
  29. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12
  30. 2025Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformerscited by 11
  31. 2025LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generationcited by 11
  32. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 11
  33. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 9
  34. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 8
  35. 2025DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacecited by 8
  36. 2025Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulationcited by 8
  37. 2026Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillationcited by 8
  38. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 8

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
  2. 2026Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation
  3. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
  4. 2026MegaSlide-DiT: Memory-Centric Adaptation and Deformable Local Attention for Efficient Video Diffusion
  5. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
  6. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
  7. 2025Bidirectional Sparse Attention for Faster Video Diffusion Training
  8. 2026RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers