mlsanityData to 7 Oct 2026

DiTFastAttn: Attention Compression for Diffusion Transformer Models

arXiv 2406.08552 ↗First posted 12 Jun 2024cs.CVcited by 109

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT. We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) Window Attention with Residual Sharing to reduce spatial redundancy; (2) Attention Sharing across Timesteps to exploit the similarity between steps; (3) Attention Sharing across CFG to skip redundant computations during conditional generation. We apply DiTFastAttn to DiT, PixArt-Sigma for image generation tasks, and OpenSora for video generation tasks. Our results show that for image generation, our method reduces up to 76% of the attention FLOPs and achieves up to 1.8x end-to-end speedup at high-resolution (2k x 2k) generation.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 34 of its 41 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

34 of 41 references.

References · 34

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2016Improved Techniques for Training GANscited by 2,938
22 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2022On Distillation of Guided Diffusion Modelscited by 721
  8. 2017Generative Adversarial Networks: An Overviewcited by 588
  9. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  10. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  11. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  12. 2022Post-training Quantization on Diffusion Modelscited by 262
  13. 2022Neighborhood Attention Transformercited by 235
  14. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  15. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  16. 2023OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Modelscited by 44
  17. 2022Vision Transformer with Super Token Samplingcited by 43
  18. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantizationcited by 42
  19. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  20. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  21. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  22. 2021PSViT: Better Vision Transformer via Token Pooling and Attention Sharingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 60; 2026: 41.

109 papers.

Sort:

By area:

  1. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  2. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  3. 2025From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeerscited by 124
  4. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  5. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  6. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 55
  7. 2025VMoBA: Mixture-of-Block Attention for Video Diffusion Modelscited by 35
  8. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 35
  9. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  10. 2025Region-Adaptive Sampling for Diffusion Transformerscited by 31
  11. 2025SpeCa: Accelerating Diffusion Transformers with Speculative Feature Cachingcited by 27
  12. 2024SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Devicecited by 25
38 more
  1. 2025Training-free Diffusion Acceleration with Bottleneck Samplingcited by 25
  2. 2025Forecast then Calibrate: Feature Caching as ODE for Efficient Diffusion Transformerscited by 23
  3. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 19
  4. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  5. 2026Flow caching for autoregressive video generationcited by 19
  6. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  7. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  8. 2025Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Cachingcited by 17
  9. 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Accelerationcited by 16
  10. 2024LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantizationcited by 16
  11. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 16
  12. 2025Analysis of Attention in Video Diffusion Transformerscited by 15
  13. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 14
  14. 2025UniCP: A Unified Caching and Pruning Framework for Efficient Video Generationcited by 13
  15. 2025Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformerscited by 13
  16. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  17. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  18. 2025ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusioncited by 11
  19. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  20. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10
  21. 2026SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Modelscited by 10
  22. 2024Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Studycited by 9
  23. 2024SparseDiT: Token Sparsification for Efficient Diffusion Transformercited by 9
  24. 2025Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformerscited by 9
  25. 2025SADA: Stability-guided Adaptive Diffusion Accelerationcited by 9
  26. 2025LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generationcited by 9
  27. 2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
  28. 2025Let Features Decide Their Own Solvers: Hybrid Feature Caching for Diffusion Transformerscited by 8
  29. 2025RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancycited by 7
  30. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 7
  31. 2025SpotEdit: Selective Region Editing in Diffusion Transformerscited by 7
  32. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformercited by 7
  33. 2025Foresight: Adaptive Layer Reuse for Accelerated and High-Quality Text-to-Video Generationcited by 6
  34. 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformercited by 6
  35. 2025RegionE: Adaptive Region-Aware Generation for Efficient Image Editingcited by 6
  36. 2026DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generationcited by 6
  37. 2025Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Modelcited by 5
  38. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
  2. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
  3. 2026Parameterized Stripe Attention for Efficient Video Generation
  4. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers
  5. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
  6. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
  7. 2025DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling
  8. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation