mlsanityData to 7 Oct 2026

A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training

arXiv 2405.17403 ↗First posted 27 May 2024cs.LG cs.AIcited by 30

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Training diffusion models is always a computation-intensive task. In this paper, we introduce a novel speed-up method for diffusion model training, called, which is based on a closer look at time steps. Our key findings are: i) Time steps can be empirically divided into acceleration, deceleration, and convergence areas based on the process increment. ii) These time steps are imbalanced, with many concentrated in the convergence area. iii) The concentrated steps provide limited benefits for diffusion training. To address this, we design an asymmetric sampling strategy that reduces the frequency of steps from the convergence area while increasing the sampling probability for steps from other areas. Additionally, we propose a weighting strategy to emphasize the importance of time steps with rapid-change process increments. As a plug-and-play and architecture-agnostic approach, SpeeD consistently achieves 3-times acceleration across various diffusion architectures, datasets, and tasks. Notably, due to its simple design, our approach significantly reduces the cost of diffusion model training with minimal overhead. Our research enables more researchers to train diffusion models at a lower cost.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 60 of its 79 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

60 of 79 references.

Closest ancestors · 26

Same direction, newest first

  1. 2024Denoising Task Difficulty-based Curriculum for Training Diffusion Models15 Mar
  2. 2024Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon14 Mar
  3. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts14 Mar
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  5. 2024Neural Network Diffusion20 Feb
  6. 2023Unmasking Bias in Diffusion Model Training12 Oct
  7. 2023Denoising Task Routing for Diffusion Models11 Oct
  8. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  9. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  10. 2023Fast Diffusion Model12 Jun
  11. 2023Addressing Negative Transfer in Diffusion Models1 Jun
  12. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models25 Apr
14 more
  1. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  2. 2023Efficient Diffusion Training via Min-SNR Weighting Strategy16 Mar
  3. 2023Consistency Models2 Mar
  4. 2022Scalable Diffusion Models with Transformers19 Dec
  5. 2022Pyramidal Denoising Diffusion Probabilistic Models3 Aug
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  8. 2022Perception Prioritized Training of Diffusion Models1 Apr
  9. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  13. 2020Denoising Diffusion Implicit Models6 Oct
  14. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2016Improved Techniques for Training GANscited by 2,938
22 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2022Video Diffusion Modelscited by 2,280
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  7. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  10. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  13. 2021Dynamic Neural Networks: A Surveycited by 405
  14. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  15. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  16. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  17. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  18. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  19. 2022Equivariant 3D-Conditional Diffusion Models for Molecular Linker Designcited by 121
  20. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  21. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  22. 2023EnfoMax: Domain Entropy and Mutual Information Maximization for Domain Generalized Face Anti-spoofingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 16; 2026: 10.

30 papers.

Sort:

By area:

  1. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  2. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 33
  3. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  4. 2025RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wildcited by 28
  5. 2025Test-Time Scaling of Diffusion Models via Noise Trajectory Searchcited by 28
  6. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformerscited by 21
  7. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
  8. 2024Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignmentcited by 14
  9. 2025Entropic Time Schedulers for Generative Diffusion Modelscited by 13
  10. 2024Adaptive Non-uniform Timestep Sampling for Accelerating Diffusion Model Trainingcited by 8
  11. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  12. 2025Efficient Multimodal Dataset Distillation via Generative Modelscited by 6
18 more
  1. 2025Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Trainingcited by 5
  2. 2025Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perceptioncited by 4
  3. 2025Improved Immiscible Diffusion: Accelerate Diffusion Training by Reducing Its Miscibilitycited by 4
  4. 2025Understanding, Accelerating, and Improving MeanFlow Trainingcited by 3
  5. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3
  6. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  7. 2025DMol: A Highly Efficient and Chemical Motif-Preserving Molecule Generation Platformcited by 1
  8. 2025Incorporating Pre-trained Diffusion Models in Solving the Schr"odinger Bridge Problemcited by 1
  9. 2026Learning Discrete Diffusion of Graphs via Free-Energy Gradient Flowscited by 1
  10. 2025Disentanglement in T-space for Faster and Distributed Training of Diffusion Models with Fewer Latent-statescited by 0
  11. 2026Gradient-Aligned Calibration for Post-Training Quantization of Diffusion Modelscited by 0
  12. 2026Stage-adaptive audio diffusion modelingcited by 0
  13. 2026Elucidating Representation Degradation Problem in Diffusion Model Trainingcited by 0
  14. 2026Timestep Rescheduling in Diffusion Inversioncited by 0
  15. 2026CenSynCMB: Centre Maps and Physics-Guided Synthesis for Microbleed Detectioncited by 0
  16. 2026TASQ: Temporal-Adaptive Bit Sparsification Quantization for Diffusion Modelscited by 0
  17. 2026Timestep Weighting: A Hidden Key to Effective ELBO-Based Flow-Matching RLcited by 0
  18. 2026SALD: Self-Referenced Advantage Learning for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Timesteps Samplers and Prediction Types
  2. 2025Pretrained Diffusion Models Are Inherently Skipped-Step Samplers
  3. 2025ETC: training-free diffusion models acceleration with Error-aware Trend Consistency
  4. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration
  5. 2024Accelerating Convergence of Score-Based Diffusion Models, Provably
  6. 2025Glance: Accelerating Diffusion Models with 1 Sample
  7. 2024Provable Acceleration for Diffusion Models under Minimal Assumptions
  8. 2025Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch