mlsanityData to 14 Oct 2026

Progressive Distillation for Fast Sampling of Diffusion Models

arXiv 2202.00512 ↗First posted 1 Feb 2022cs.LG cs.AI stat.MLcited by 69

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models have recently shown great promise for generative modeling, outperforming GANs on perceptual quality and autoregressive models at density estimation. A remaining downside is their slow sampling time: generating high quality samples takes many hundreds or thousands of model evaluations. Here we make two contributions to help eliminate this downside: First, we present new parameterizations of diffusion models that provide increased stability when using few sampling steps. Second, we present a method to distill a trained deterministic diffusion sampler, using many steps, into a new diffusion model that takes half as many sampling steps. We then keep progressively applying this distillation procedure to our model, halving the number of required sampling steps each time. On standard image generation benchmarks like CIFAR-10, ImageNet, and LSUN, we start out with state-of-the-art samplers taking as many as 8192 steps, and are able to distill down to models taking as few as 4 steps without losing much perceptual quality; achieving, for example, a FID of 3.0 on CIFAR-10 in 4 steps. Finally, we show that the full progressive distillation procedure does not take more time than it takes to train the original model, thus representing an efficient solution for generative modeling using diffusion at both train and test time.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 23 of its 36 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

23 of 36 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Bilateral Denoising Diffusion Models26 Aug
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  3. 2021Variational Diffusion Models1 Jul
  4. 2021Non Gaussian Denoising Diffusion Models14 Jun
  5. 2021Score-based Generative Modeling in Latent Space10 Jun
  6. 2021Learning to Efficiently Sample from Diffusion Probabilistic Models7 Jun
  7. 2021On Fast Sampling of Diffusion Probabilistic Models31 May
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  9. 2021Gotta Go Fast When Generating Data with Score-Based Models28 May
  10. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  11. 2021Noise Estimation for Generative Diffusion Models6 Apr
  12. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
1 more
  1. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan

Other references · 10

  1. 2017Decoupled Weight Decay Regularizationcited by 1,831
  2. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  3. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  4. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  5. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  6. 2020Permutation Invariant Graph Generation via Score-Based Generative Modelingcited by 270
  7. 2020Learning Gradient Fields for Shape Generationcited by 245
  8. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
  9. 2017Fast Generation for Convolutional Autoregressive Modelscited by 14
  10. 2020Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminatorcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 22; 2025: 19; 2026: 26.

69 papers.

Sort:

By area:

  1. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  2. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 88
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  4. 2024Phased Consistency Modelscited by 85
  5. 2024Tutorial on Diffusion Models for Imaging and Visioncited by 49
  6. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 48
  7. 2024Foundation Models for Music: A Surveycited by 37
  8. 2024Token Pruning for Caching Better: 9 Times Acceleration on Stable Diffusion for Freecited by 30
  9. 2026Flash-WAM: Modality-Aware Distillation for World Action Modelscited by 19
  10. 2024One Step Diffusion-based Super-Resolution with Time-Aware Distillationcited by 17
  11. 2025Accelerating Diffusion Transformer via Gradient-Optimized Cachecited by 15
  12. 2026Alaya-EVOKE: From Linear-Scaling Supervision to Endless Worldcited by 14
38 more
  1. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 13
  2. 2025Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithmscited by 12
  3. 2024Accelerating Video Diffusion Models via Distribution Matchingcited by 11
  4. 2024Stable Consistency Tuning: Understanding and Improving Consistency Modelscited by 9
  5. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  6. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 9
  7. 2024Tri-Ergon: Fine-grained Video-to-Audio Generation with Multi-modal Conditions and LUFS Controlcited by 8
  8. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  9. 2024MoViE: Mobile Diffusion for Video Editingcited by 6
  10. 2025Scale Where It Matters: Training-Free Localized Scaling for Diffusion Modelscited by 6
  11. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  12. 2026GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulationcited by 5
  13. 2023Modeling Molecular Structures with Intrinsic Diffusion Modelscited by 4
  14. 2023Simultaneous Image-to-Zero and Zero-to-Noise: Diffusion Models with Analytical Image Attenuationcited by 4
  15. 2025Realism Control One-step Diffusion for Real-World Image Super-Resolutioncited by 4
  16. 20261.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillationcited by 4
  17. 2024Variational Flow Models: Flowing in Your Stylecited by 3
  18. 2024From Noise to Nuance: Advances in Deep Generative Image Modelscited by 3
  19. 2025A Novel Diffusion Model for Pairwise Geoscience Data Generation with Unbalanced Training Datasetcited by 3
  20. 2025From Structure to Detail: Hierarchical Distillation for Efficient Diffusion Modelcited by 3
  21. 2026Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editingcited by 3
  22. 2024Hybrid diffusion models: combining supervised and generative pretraining for label-efficient fine-tuning of segmentation modelscited by 2
  23. 2024RAIN: Real-time Animation of Infinite Video Streamcited by 2
  24. 2025FEB-Cache: Frequency-Guided Exposure Bias Reduction for Enhancing Diffusion Transformer Cachingcited by 2
  25. 2025Foundations of Diffusion Models in General State Spaces: A Self-Contained Introductioncited by 2
  26. 2026Variance Reduction for Expectations with Diffusion Teacherscited by 2
  27. 2024E2GAN: Efficient Training of Efficient GANs for Image-to-Image Translationcited by 1
  28. 2025DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generationcited by 1
  29. 2025DT-NVS: Diffusion Transformers for Novel View Synthesiscited by 1
  30. 2025Verifier Threshold: An Efficient Test-Time Scaling Approach for Image Generationcited by 1
  31. 2026FARI: Robust One-Step Inversion for Watermarking in Diffusion Modelscited by 1
  32. 2026FlowErase-OPD: Multi-Concept Erasure via Anchored On-Policy Distillation in Flow Matching Modelscited by 1
  33. 2024Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Modelcited by 0
  34. 2024IMPROVE: Improving Medical Plausibility without Reliance on HumanValidation -- An Enhanced Prototype-Guided Diffusion Frameworkcited by 0
  35. 2024BudgetFusion: Perceptually-Guided Adaptive Diffusion Modelscited by 0
  36. 2024Generative Diffusion Modeling: A Practical Handbookcited by 0
  37. 2024E2ED^2:Direct Mapping from Noise to Data for Enhanced Diffusion Modelscited by 0
  38. 2025Distillation-Driven Diffusion Model for Multi-Scale MRI Super-Resolution: Make 1.5T MRI Great Againcited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
  2. 2024Simple and Fast Distillation of Diffusion Models
  3. 2025Adaptive Destruction Processes for Diffusion Samplers
  4. 2023Adversarial Diffusion Distillation
  5. 2025A Survey on Pre-Trained Diffusion Model Distillations
  6. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
  7. 2024Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion Sampling
  8. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models