mlsanityData to 7 Oct 2026

Multistep Distillation of Diffusion Models via Moment Matching

arXiv 2406.04103 ↗First posted 6 Jun 2024cs.LG cs.AI cs.CVcited by 76

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We present a new method for making diffusion models faster to sample. The method distills many-step diffusion models into few-step models by matching conditional expectations of the clean data given noisy data along the sampling trajectory. Our approach extends recently proposed one-step methods to the multi-step case, and provides a new perspective by interpreting these approaches in terms of moment matching. By using up to 8 sampling steps, we obtain distilled models that outperform not only their one-step versions but also their original many-step teacher models, obtaining new state-of-the-art results on the Imagenet dataset. We also show promising results on a large text-to-image model where we achieve fast generation of high resolution images directly in image space, without needing autoencoders or upsamplers.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 31 of its 43 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

31 of 43 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Multistep Consistency Models11 Mar
  2. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  3. 2023Improved Techniques for Training Consistency Models22 Oct
  4. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  5. 2023Semi-Implicit Denoising Diffusion Models (SIDDMs)21 Jun
  6. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models29 May
  7. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation7 Mar
  8. 2023Consistency Models2 Mar
  9. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  10. 2022Classifier-Free Diffusion Guidance26 Jul
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  12. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
5 more
  1. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  2. 2021Variational Diffusion Models1 Jul
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  4. 2020Denoising Diffusion Implicit Models6 Oct
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  9. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  10. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  11. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  12. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
2 more
  1. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109
  2. 2024PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Acceleratorcited by 95

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 39; 2026: 24.

76 papers.

Sort:

By area:

  1. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  2. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  3. 2025Inductive Moment Matchingcited by 107
  4. 2025How to build a consistency model: Learning flow maps via self-distillationcited by 96
  5. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 85
  6. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 81
  7. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  8. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusioncited by 74
  9. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  10. 2025Learning Few-Step Diffusion Models by Trajectory Distribution Matchingcited by 49
  11. 2025One-step Diffusion Models with f-Divergence Distribution Matchingcited by 43
  12. 2025Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesiscited by 39
38 more
  1. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30
  2. 2025Accelerated Diffusion Models via Speculative Samplingcited by 30
  3. 2024Flow Generator Matchingcited by 26
  4. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 25
  5. 2025Distributional Diffusion Models with Scoring Rulescited by 25
  6. 2025pi-Flow: Policy-Based Few-Step Generation via Imitation Distillationcited by 25
  7. 2024Truncated Consistency Modelscited by 21
  8. 2025Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generationcited by 19
  9. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  10. 2026One-Step Generative Modeling via Wasserstein Gradient Flowscited by 18
  11. 2024Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matchingcited by 13
  12. 2025Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instructioncited by 13
  13. 2025Flow Map Distillation Without Datacited by 13
  14. 2025RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectoriescited by 12
  15. 2025Learn to Guide Your Diffusion Modelcited by 12
  16. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  17. 2025Adversarial Flow Modelscited by 10
  18. 2026Representation Distribution Matching for One-Step Visual Generationcited by 10
  19. 2024Multi-student Diffusion Distillation for Better One-step Generatorscited by 9
  20. 2025Learning to Integrate Diffusion ODEs by Averaging the Derivativescited by 9
  21. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-trainingcited by 8
  22. 2025DiffRatio: Training One-Step Diffusion Models Without Teacher Supervisioncited by 8
  23. 2026Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMDcited by 8
  24. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Rewardcited by 7
  25. 2026Adaptive Video Distillation: Mitigating Oversaturation and Temporal Collapse in Few-Step Generationcited by 7
  26. 2024See Further When Clear: Curriculum Consistency Modelcited by 6
  27. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  28. 2026Adaptive Spectral Feature Forecasting for Diffusion Sampling Accelerationcited by 5
  29. 2026Spectrally-Guided Diffusion Noise Schedulescited by 5
  30. 2025Fast Autoregressive Models for Continuous Latent Generationcited by 4
  31. 2025On the Design of One-step Diffusion via Shortcutting Flow Pathscited by 4
  32. 2026DisCa: Accelerating Video Diffusion Transformers with Distillation-Compatible Learnable Feature Cachingcited by 4
  33. 2026Distribution Matching Distillation without Fake Score Networkcited by 4
  34. 2025Generating ensembles of spatially-coherent in-situ forecasts using flow matchingcited by 3
  35. 2025Understanding, Accelerating, and Improving MeanFlow Trainingcited by 3
  36. 2025Self-Evaluation Unlocks Any-Step Text-to-Image Generationcited by 3
  37. 2026Forward-Learned Discrete Diffusion: Learning how to noise to denoise fastercited by 3
  38. 2025Geometric Moment Alignment for Domain Adaptation via Siegel Embeddingscited by 2

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scale-wise Distillation of Diffusion Models
  2. 2022On Distillation of Guided Diffusion Models
  3. 2022Progressive Distillation for Fast Sampling of Diffusion Models
  4. 2024SFDDM: Single-fold Distillation for Diffusion models
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications
  6. 2024A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models
  7. 2025Glance: Accelerating Diffusion Models with 1 Sample
  8. 2024One Step Diffusion via Shortcut Models