mlsanityData to 7 Oct 2026

Analyzing and Improving the Training Dynamics of Diffusion Models

arXiv 2312.02696 ↗First posted 5 Dec 2023cs.CV cs.AI cs.LGcited by 396

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models currently dominate the field of data-driven image synthesis with their unparalleled scaling to large datasets. In this paper, we identify and rectify several causes for uneven and ineffective training in the popular ADM diffusion model architecture, without altering its high-level structure. Observing uncontrolled magnitude changes and imbalances in both the network activations and weights over the course of training, we redesign the network layers to preserve activation, weight, and update magnitudes on expectation. We find that systematic application of this philosophy eliminates the observed drifts and imbalances, resulting in considerably better networks at equal computational complexity. Our modifications improve the previous record FID of 2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic sampling. As an independent contribution, we present a method for setting the exponential moving average (EMA) parameters post-hoc, i.e., after completing the training run. This allows precise tuning of EMA length without the cost of performing several training runs, and reveals its surprising interactions with network architecture, training time, and guidance.

Key facts. Cited by 396 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 77 of its 97 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

77 of 97 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  2. 2022Scalable Diffusion Models with Transformers19 Dec
  3. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications2 Sep
  4. 2022Classifier-Free Diffusion Guidance26 Jul
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  11. 2020Denoising Diffusion Implicit Models6 Oct
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  13. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  14. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  15. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  16. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  17. 2018Demystifying MMD GANscited by 1,268
  18. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  19. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  20. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  21. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  22. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  23. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  24. 2017Self-Normalizing Neural Networkscited by 591
  25. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  26. 2017Large Batch Training of Convolutional Networkscited by 529
  27. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  28. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  29. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  30. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  31. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  32. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  33. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  34. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  35. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  36. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  37. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  38. 2018Three Mechanisms of Weight Decay Regularizationcited by 112

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 90; 2025: 195; 2026: 108.

396 papers.

Sort:

By area:

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  3. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  4. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  5. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  6. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  7. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  8. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  9. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 240
  10. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 209
  11. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  12. 2025History-Guided Video Diffusioncited by 165
38 more
  1. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  2. 2024Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealingcited by 157
  3. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  4. 2024Consistency Models Made Easycited by 140
  5. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  6. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  7. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  8. 2025Inductive Moment Matchingcited by 107
  9. 2025How to build a consistency model: Learning flow maps via self-distillationcited by 96
  10. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  11. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 85
  12. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 81
  13. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  14. 2024Classifier-Free Guidance is a Predictor-Correctorcited by 79
  15. 2024Distilling Diffusion Models into Conditional GANscited by 78
  16. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  17. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
  18. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusioncited by 74
  19. 2025Improving the Diffusability of Autoencoderscited by 74
  20. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  21. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  22. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  23. 2024No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Modelscited by 52
  24. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  25. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47
  26. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Spacecited by 44
  27. 2025AlphaFlow: Understanding and Improving MeanFlow Modelscited by 40
  28. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  29. 2024Improved Noise Schedule for Diffusion Trainingcited by 38
  30. 2026Categorical Flow Mapscited by 37
  31. 2024Language-Guided Image Tokenization for Generationcited by 36
  32. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  33. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenizationcited by 35
  34. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  35. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  36. 2024Simple ReFlow: Improved Techniques for Fast Flow Modelscited by 34
  37. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  38. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 33

50 most cited of 396 All 396 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective
  2. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
  3. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning
  4. 2024A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models
  5. 2023Neural Diffusion Models
  6. 2025Glance: Accelerating Diffusion Models with 1 Sample
  7. 2025CacheQuant: Comprehensively Accelerated Diffusion Models
  8. 2023AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation