mlsanityData to 7 Oct 2026

Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed

arXiv 2101.02388 ↗First posted 7 Jan 2021cs.LGcited by 344

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Iterative generative models, such as noise conditional score networks and denoising diffusion probabilistic models, produce high quality samples by gradually denoising an initial noise vector. However, their denoising process has many steps, making them 2-3 orders of magnitude slower than other generative models such as GANs and VAEs. In this paper, we establish a novel connection between knowledge distillation and image generation with a technique that distills a multi-step denoising process into a single step, resulting in a sampling speed similar to other single-step generative models. Our Denoising Student generates high quality samples comparable to GANs on the CIFAR-10 and CelebA datasets, without adversarial training. We demonstrate that our method scales to higher resolutions through experiments on 256 x 256 LSUN. Code and checkpoints are available at https://github.com/tcl9876/Denoising_Student

Key facts. Cited by 344 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 33 of its 47 references that are in the map. First posted on arXiv 7 Jan 2021.

Builds on

33 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Implicit Models6 Oct
  3. 2020Denoising Diffusion Probabilistic Models19 Jun
  4. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
17 more
  1. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  2. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  5. 2019Contrastive Representation Distillationcited by 734
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2018Born Again Neural Networkscited by 539
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  10. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  11. 2016Learning in Implicit Generative Modelscited by 124
  12. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  13. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
  14. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  15. 2020Improving GAN Training with Probability Ratio Clipping and Sample Reweightingcited by 18
  16. 2020Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognitioncited by 6
  17. 2018Learning Implicit Generative Models with the Method of Learned Momentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 34; 2023: 72; 2024: 87; 2025: 93; 2026: 49.

344 papers.

Sort:

By area:

  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  5. 2023Consistency Modelscited by 1,778
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  9. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  10. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  11. 2022On Distillation of Guided Diffusion Modelscited by 721
  12. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
38 more
  1. 2021Score-based Generative Modeling in Latent Spacecited by 634
  2. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
  3. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  4. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  5. 2023Improved Techniques for Training Consistency Modelscited by 430
  6. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  7. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  8. 2024One Step Diffusion via Shortcut Modelscited by 373
  9. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  10. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  11. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  12. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  13. 2022Post-training Quantization on Diffusion Modelscited by 262
  14. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  15. 2022A Survey on Generative Diffusion Modelcited by 238
  16. 2023SinSR: Diffusion-Based Image Super-Resolution in a Single Stepcited by 235
  17. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  18. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  19. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  20. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 211
  21. 2022Diffusion-GAN: Training GANs with Diffusioncited by 203
  22. 2022Fast Sampling of Diffusion Models via Operator Learningcited by 173
  23. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  24. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  25. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 149
  26. 2024Consistency Models Made Easycited by 140
  27. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  28. 2022gDDIM: Generalized denoising diffusion implicit modelscited by 125
  29. 2025The Diffusion Dualitycited by 124
  30. 2022Diffusion Probabilistic Model Made Slimcited by 121
  31. 2022Wavelet Diffusion Models are fast and scalable Image Generatorscited by 116
  32. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillationcited by 115
  33. 2024One-Step Image Translation with Text-to-Image Modelscited by 111
  34. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109
  35. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  36. 2025Inductive Moment Matchingcited by 107
  37. 2024Improving the Training of Rectified Flowscited by 106
  38. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 103

50 most cited of 344 All 344 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Denoising Score Distillation: From Noisy Diffusion Pretraining to One-Step High-Quality Generation
  2. 2025Noise Conditional Variational Score Distillation
  3. 2023KD-DLGAN: Data Limited Image Generation via Knowledge Distillation
  4. 2025Generative Dataset Distillation using Min-Max Diffusion Model
  5. 2023Adversarial Diffusion Distillation
  6. 2023DiffFlow: A Unified SDE Framework for Score-Based Diffusion Models and Generative Adversarial Networks
  7. 2025Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression
  8. 2024Evaluating the design space of diffusion-based generative models