# Progressive Distillation for Fast Sampling of Diffusion Models

arXiv: https://arxiv.org/abs/2202.00512 · HTML: https://mlsanity.com/paper/2202.00512/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 23 of 23)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021)
- [Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed](https://mlsanity.com/paper/2101.02388/) (2021)
- [Permutation Invariant Graph Generation via Score-Based Generative Modeling](https://mlsanity.com/paper/2003.00638/) (2020)
- [Learning Gradient Fields for Shape Generation](https://mlsanity.com/paper/2008.06520/) (2020)
- [Gotta Go Fast When Generating Data with Score-Based Models](https://mlsanity.com/paper/2105.14080/) (2021)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [On Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.00132/) (2021)
- [Learning to Efficiently Sample from Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.03802/) (2021)
- [Noise Estimation for Generative Diffusion Models](https://mlsanity.com/paper/2104.02600/) (2021)
- [Non Gaussian Denoising Diffusion Models](https://mlsanity.com/paper/2106.07582/) (2021)
- [Bilateral Denoising Diffusion Models](https://mlsanity.com/paper/2108.11514/) (2021)
- [Fast Generation for Convolutional Autoregressive Models](https://mlsanity.com/paper/1704.06001/) (2017)
- [Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator](https://mlsanity.com/paper/2004.01704/) (2020)

## Built on by (most cited first, 30 of 69)

- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review](https://mlsanity.com/paper/2501.09685/) (2025)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Phased Consistency Models](https://mlsanity.com/paper/2405.18407/) (2024)
- [Tutorial on Diffusion Models for Imaging and Vision](https://mlsanity.com/paper/2403.18103/) (2024)
- [Multimodal Latent Language Modeling with Next-Token Diffusion](https://mlsanity.com/paper/2412.08635/) (2024)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Token Pruning for Caching Better: 9 Times Acceleration on Stable Diffusion for Free](https://mlsanity.com/paper/2501.00375/) (2024)
- [Flash-WAM: Modality-Aware Distillation for World Action Models](https://mlsanity.com/paper/2606.05254/) (2026)
- [One Step Diffusion-based Super-Resolution with Time-Aware Distillation](https://mlsanity.com/paper/2408.07476/) (2024)
- [Accelerating Diffusion Transformer via Gradient-Optimized Cache](https://mlsanity.com/paper/2503.05156/) (2025)
- [Alaya-EVOKE: From Linear-Scaling Supervision to Endless World](https://mlsanity.com/paper/2608.13546/) (2026)
- [S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation](https://mlsanity.com/paper/2508.04016/) (2025)
- [Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithms](https://mlsanity.com/paper/2503.07154/) (2025)
- [Accelerating Video Diffusion Models via Distribution Matching](https://mlsanity.com/paper/2412.05899/) (2024)
- [Stable Consistency Tuning: Understanding and Improving Consistency Models](https://mlsanity.com/paper/2410.18958/) (2024)
- [Text-to-Image Synthesis: A Decade Survey](https://mlsanity.com/paper/2411.16164/) (2024)
- [Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition](https://mlsanity.com/paper/2510.01068/) (2025)
- [Tri-Ergon: Fine-grained Video-to-Audio Generation with Multi-modal Conditions and LUFS Control](https://mlsanity.com/paper/2412.20378/) (2024)
- [V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think](https://mlsanity.com/paper/2604.23380/) (2026)
- [MoViE: Mobile Diffusion for Video Editing](https://mlsanity.com/paper/2412.06578/) (2024)
- [Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models](https://mlsanity.com/paper/2511.19917/) (2025)
- [Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator](https://mlsanity.com/paper/2503.15457/) (2025)
- [GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2609.05588/) (2026)
- [Modeling Molecular Structures with Intrinsic Diffusion Models](https://mlsanity.com/paper/2302.12255/) (2023)
- [Simultaneous Image-to-Zero and Zero-to-Noise: Diffusion Models with Analytical Image Attenuation](https://mlsanity.com/paper/2306.13720/) (2023)
- [Realism Control One-step Diffusion for Real-World Image Super-Resolution](https://mlsanity.com/paper/2509.10122/) (2025)
- [1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation](https://mlsanity.com/paper/2604.04018/) (2026)
- [Variational Flow Models: Flowing in Your Style](https://mlsanity.com/paper/2402.02977/) (2024)
- [From Noise to Nuance: Advances in Deep Generative Image Models](https://mlsanity.com/paper/2412.09656/) (2024)

## Abstract (arXiv)

Diffusion models have recently shown great promise for generative modeling, outperforming GANs on perceptual quality and autoregressive models at density estimation. A remaining downside is their slow sampling time: generating high quality samples takes many hundreds or thousands of model evaluations. Here we make two contributions to help eliminate this downside: First, we present new parameterizations of diffusion models that provide increased stability when using few sampling steps. Second, we present a method to distill a trained deterministic diffusion sampler, using many steps, into a new diffusion model that takes half as many sampling steps. We then keep progressively applying this distillation procedure to our model, halving the number of required sampling steps each time. On standard image generation benchmarks like CIFAR-10, ImageNet, and LSUN, we start out with state-of-the-art samplers taking as many as 8192 steps, and are able to distill down to models taking as few as 4 steps without losing much perceptual quality; achieving, for example, a FID of 3.0 on CIFAR-10 in 4 steps. Finally, we show that the full progressive distillation procedure does not take more time than it takes to train the original model, thus representing an efficient solution for generative modeling using diffusion at both train and test time.