mlsanityData to 7 Oct 2026

DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers

arXiv 2503.22796 ↗First posted 28 Mar 2025cs.CV cs.AIcited by 19

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-to-image generation models, especially Multimodal Diffusion Transformers (MMDiT), have shown remarkable progress in generating high-quality images. However, these models often face significant computational bottlenecks, particularly in attention mechanisms, which hinder their scalability and efficiency. In this paper, we introduce DiTFastAttnV2, a post-training compression method designed to accelerate attention in MMDiT. Through an in-depth analysis of MMDiT's attention patterns, we identify key differences from prior DiT-based methods and propose head-wise arrow attention and caching mechanisms to dynamically adjust attention heads, effectively bridging this gap. We also design an Efficient Fused Kernel for further acceleration. By leveraging local metric methods and optimization techniques, our approach significantly reduces the search time for optimal compression schemes to just minutes while maintaining generation quality. Furthermore, with the customized kernel, DiTFastAttnV2 achieves a 68% reduction in attention FLOPs and 1.5x end-to-end speedup on 2K image generation without compromising visual fidelity.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

51 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Fast Video Generation with Sliding Tile Attention6 Feb
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity3 Feb
  3. 2024LTX-Video: Realtime Video Latent Diffusion30 Dec
  4. 2024CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up20 Dec
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  6. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Models12 Jun
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
32 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  8. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  9. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  10. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  13. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  14. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  15. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  16. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  17. 2022Post-training Quantization on Diffusion Modelscited by 262
  18. 2022Neighborhood Attention Transformercited by 235
  19. 2023Token Merging for Fast Stable Diffusioncited by 220
  20. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 169
  21. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  22. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  23. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 149
  24. 2024FORA: Fast-Forward Caching in Diffusion Transformer Accelerationcited by 127
  25. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  26. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  27. 2024LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insightscited by 65
  28. 2024LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Modelscited by 52
  29. 2023FRDiff : Feature Reuse for Universal Training-free Acceleration of Diffusion Modelscited by 19
  30. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  31. 2024ToDo: Token Downsampling for Efficient Generation of High-Resolution Imagescited by 14
  32. 2024Structured Initialization for Attention in Vision Transformerscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  2. 2025Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformerscited by 11
  3. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  4. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10
  5. 2025SADA: Stability-guided Adaptive Diffusion Accelerationcited by 9
  6. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 9
  7. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformercited by 7
  8. 2025RegionE: Adaptive Region-Aware Generation for Efficient Image Editingcited by 6
  9. 2025Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondencecited by 4
  10. 2026ZEUS: Accelerating Diffusion Models with Only Second-Order Predictorcited by 3
  11. 2025FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformerscited by 2
  12. 2025DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibrationcited by 1
7 more
  1. 2025UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attentioncited by 1
  2. 2026NanoFLUX: Distillation-Driven Compression of Large Text-to-Image Generation Models for Mobile Devicescited by 0
  3. 20266Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Modelscited by 0
  4. 2026HierEdit: Region-Aware Hierarchical Diffusion for Efficient High-Resolution Editingcited by 0
  5. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Modelscited by 0
  6. 2026SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inferencecited by 0
  7. 2026HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EDiT: Efficient Diffusion Transformers with Linear Compressed Attention
  2. 2026Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation
  3. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
  4. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers
  5. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention
  6. 2026Parameterized Stripe Attention for Efficient Video Generation
  7. 2025E-MMDiT: Revisiting Multimodal Diffusion Transformer Design for Fast Image Synthesis under Limited Resources
  8. 2026DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation