mlsanityData to 7 Oct 2026

Dynamic Diffusion Transformer

arXiv 2410.03456 ↗First posted 4 Oct 2024cs.CVcited by 60

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion Transformer (DiT), an emerging diffusion model for image generation, has demonstrated superior performance but suffers from substantial computational costs. Our investigations reveal that these costs stem from the static inference paradigm, which inevitably introduces redundant computation in certain diffusion timesteps and spatial regions. To address this inefficiency, we propose Dynamic Diffusion Transformer (DyDiT), an architecture that dynamically adjusts its computation along both timestep and spatial dimensions during generation. Specifically, we introduce a Timestep-wise Dynamic Width (TDW) approach that adapts model width conditioned on the generation timesteps. In addition, we design a Spatial-wise Dynamic Token (SDT) strategy to avoid redundant computation at unnecessary spatial locations. Extensive experiments on various datasets and different-sized models verify the superiority of DyDiT. Notably, with <3% additional fine-tuning iterations, our method reduces the FLOPs of DiT-XL by 51%, accelerates generation by 1.73, and achieves a competitive FID score of 2.07 on ImageNet. The code is publicly available at https://github.com/NUS-HPC-AI-Lab/ Dynamic-Diffusion-Transformer.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 71 of its 85 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

71 of 85 references.

Closest ancestors · 32

Same direction, newest first

  1. 2024A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models12 Aug
  2. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators11 Aug
  3. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)1 Jul
  4. 2024Diffusion Models in Low-Level Vision: A Survey17 Jun
  5. 2024Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization13 Jun
  6. 2024SparseDM: Toward Sparse Efficient Diffusion Models16 Apr
  7. 2024T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching21 Feb
  8. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  9. 2024PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models10 Jan
  10. 2023DiffiT: Diffusion Vision Transformers for Image Generation4 Dec
  11. 2023DeepCache: Accelerating Diffusion Models for Free1 Dec
  12. 2023Diffusion Models Without Attention30 Nov
20 more
  1. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  2. 2023Temporal Dynamic Quantization for Diffusion Models4 Jun
  3. 2023Structural Pruning for Diffusion Models18 May
  4. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning13 Apr
  5. 2023Token Merging for Fast Stable Diffusion30 Mar
  6. 2023Consistency Models2 Mar
  7. 2023Q-Diffusion: Quantizing Diffusion Models8 Feb
  8. 2022Scalable Diffusion Models with Transformers19 Dec
  9. 2022Post-training Quantization on Diffusion Models28 Nov
  10. 2022Diffusion Probabilistic Model Made Slim27 Nov
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  12. 2022On Distillation of Guided Diffusion Models6 Oct
  13. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  14. 2022Classifier-Free Diffusion Guidance26 Jul
  15. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  16. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  17. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  18. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  19. 2020Denoising Diffusion Implicit Models6 Oct
  20. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2022Token Merging: Your ViT But Fastercited by 995
  11. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
27 more
  1. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  2. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  3. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  7. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  8. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  9. 2021Generating Images with Sparse Representationscited by 262
  10. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  11. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  12. 2021Accelerating Sparse Deep Neural Networkscited by 213
  13. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  14. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  15. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  16. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 99
  17. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  18. 2021Dynamic Slimmable Networkcited by 83
  19. 2017Fine-Grained Car Detection for Visual Census Estimationcited by 76
  20. 2021Adaptive Focus for Efficient Video Recognitioncited by 76
  21. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  22. 2022The ArtBench Dataset: Benchmarking Generative Models with Artworkscited by 64
  23. 2022Learning to Weight Samples for Dynamic Early-exiting Networkscited by 44
  24. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  25. 2023Latency-aware Unified Dynamic Networks for Efficient Image Recognitioncited by 26
  26. 2023Dynamic Grained Encoder for Vision Transformerscited by 24
  27. 2024AdaNAT: Exploring Adaptive Policy for Token-Based Image Generationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 24; 2026: 29.

60 papers.

Sort:

By area:

  1. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  2. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47
  3. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  4. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  5. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  6. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 29
  7. 2025Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Controlcited by 27
  8. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  9. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformerscited by 21
  10. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  11. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 20
  12. 2025Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Accelerationcited by 16
38 more
  1. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 13
  2. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  3. 2024Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraintscited by 11
  4. 2025Scale-wise Distillation of Diffusion Modelscited by 11
  5. 2024SparseDiT: Token Sparsification for Efficient Diffusion Transformercited by 9
  6. 2026Elastic Diffusion Transformercited by 7
  7. 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformercited by 6
  8. 2025EA-ViT: Efficient Adaptation for Elastic Vision Transformercited by 5
  9. 2025ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformerscited by 5
  10. 2025ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generationcited by 4
  11. 2025Neural-Driven Image Editingcited by 4
  12. 2025Rare Text Semantics Were Always There in Your Diffusion Transformercited by 4
  13. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  14. 2025Are Image-to-Video Models Good Zero-Shot Image Editors?cited by 4
  15. 2025Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generationcited by 3
  16. 2025Parallel Diffusion Solver via Residual Dirichlet Policy Optimizationcited by 3
  17. 2026Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classificationcited by 3
  18. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 3
  19. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3
  20. 2025Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolutioncited by 2
  21. 2026Token Pruning for In-Context Generation in Diffusion Transformerscited by 2
  22. 2026Shiva-DiT: Residual-Based Differentiable Top-k Selection for Efficient Diffusion Transformerscited by 2
  23. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsitycited by 2
  24. 2026How Far Are Video Models from True Multimodal Reasoning?cited by 2
  25. 2026CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformerscited by 2
  26. 2025DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generationcited by 1
  27. 2026Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?cited by 1
  28. 2026DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generationcited by 1
  29. 2026The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latentscited by 1
  30. 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generationcited by 1
  31. 2026BAG: Budget-Aware Gating for Diffusion Cachingcited by 1
  32. 2025Change of Thought: Adaptive Test-Time Computationcited by 0
  33. 2025Collaborative-Distilled Diffusion Models (CDDM) for Accelerated and Lightweight Trajectory Predictioncited by 0
  34. 2026RemEdit: Efficient Diffusion Editing with Riemannian Geometrycited by 0
  35. 2026StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Modelscited by 0
  36. 2026Error as Signal: Stiffness-Aware Diffusion Sampling via Embedded Runge-Kutta Guidancecited by 0
  37. 2026AdaGen: Learning Adaptive Policy for Image Synthesiscited by 0
  38. 2026LumaFlux: Lifting 8-Bit Worlds to HDR Reality with Physically-Guided Diffusion Transformerscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025D2iT: Dynamic Diffusion Transformer for Accurate Image Generation
  2. 2025DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
  3. 2025DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusion
  4. 2026From Sketch to Fresco: Efficient Diffusion Transformer with Progressive Resolution
  5. 2023AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation
  6. 2026Just-in-Time: Training-Free Spatial Acceleration for Diffusion Transformers
  7. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers
  8. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas