mlsanityData to 7 Oct 2026

AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation

arXiv 2604.18348 ↗First posted 20 Apr 2026cs.CV cs.AIcited by 5

On the map: Diffusion models → Video generation

Abstract (arXiv)

Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 49 of its 59 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

49 of 59 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation24 Jun
  2. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation24 May
  3. 2025VSA: Faster Video Diffusion with Trainable Sparse Attention19 May
  4. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generation28 Feb
  5. 2025Fast Video Generation with Sliding Tile Attention6 Feb
  6. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity3 Feb
  7. 2024LTX-Video: Realtime Video Latent Diffusion30 Dec
  8. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation27 Oct
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  10. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generation31 Jul
  11. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  12. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
4 more
  1. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videos25 Dec
  2. 2023A Survey on Video Diffusion Models16 Oct
  3. 2023ModelScope Text-to-Video Technical Report12 Aug
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  6. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  7. 2022Diffusion Models in Vision: A Surveycited by 1,008
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
21 more
  1. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  2. 2022Transformer Quality in Linear Timecited by 189
  3. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  4. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  5. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  6. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 80
  7. 2024SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfercited by 61
  8. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  9. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  10. 2019An Empirical Study of Spatial Attention Mechanisms in Deep Networkscited by 58
  11. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  12. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 39
  13. 2021Transformer Acceleration with Dynamic Sparse Attentioncited by 19
  14. 2024How Sparse Attention Approximates Exact Attention? Your Attention is Naturally nC-Sparsecited by 16
  15. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  16. 2022Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequencescited by 12
  17. 2021Dynamic Token Normalization Improves Vision Transformerscited by 10
  18. 2024IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Modelscited by 9
  19. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 7
  20. 2024PQD: Post-training Quantization for Efficient Diffusion Modelscited by 6
  21. 2025Scale-invariant Attentioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusioncited by 1
  2. 2026Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Modelscited by 1
  3. 2026SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attentioncited by 0
  4. 2026ClusterAttention: A training-free speedup of bidirectional attentioncited by 0
  5. 2026PARK: Accurate Block Retrieval for Sparse Attention in Video Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion
  2. 2026SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
  3. 2026Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering
  4. 2025Bidirectional Sparse Attention for Faster Video Diffusion Training
  5. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
  6. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
  7. 2026LVSA: Training-Free Sparse Attention for Long Video Diffusion
  8. 2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation