mlsanityData to 7 Oct 2026

Accelerating Diffusion Sampling with Classifier-based Feature Distillation

arXiv 2211.12039 ↗First posted 22 Nov 2022cs.CVcited by 15

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of N-step teacher sampler with N/2-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed Classifier-based Feature Distillation (CFD). Instead of aligning output images, we distill teacher's sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance. We also introduce a dataset-oriented loss to further optimize the model. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling. Code is provided at urlhttps://github.com/zju-SWJ/RCFD.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 23 of its 29 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

23 of 29 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  2. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  3. 2021On Fast Sampling of Diffusion Probabilistic Models31 May
  4. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  5. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  6. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  7. 2020Denoising Diffusion Implicit Models6 Oct
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2018Born Again Neural Networkscited by 539
  9. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
  10. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  11. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  12. 2019Online Knowledge Distillation with Diverse Peerscited by 127
3 more
  1. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  2. 2020Generative Adversarial Networks (GANs Survey): Challenges, Solutions, and Future Directionscited by 65
  3. 2021BIGRoC: Boosting Image Generation via a Robust Classifiercited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 3; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  2. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Modelscited by 45
  3. 2024Simple and Fast Distillation of Diffusion Modelscited by 31
  4. 2024Towards a mathematical theory for consistency training in diffusion modelscited by 25
  5. 2023Catch-Up Distillation: You Only Need to Train Once for Accelerating Samplingcited by 13
  6. 2024Knowledge Distillation with Refined Logitscited by 9
  7. 2024Accelerating Diffusion Models with One-to-Many Knowledge Distillationcited by 9
  8. 2025FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Trainingcited by 5
  9. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  10. 2024Theoretical research on generative diffusion models: an overviewcited by 2
  11. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  12. 2024Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillationcited by 1
3 more
  1. 2024Knowledge Translation: A New Pathway for Model Compressioncited by 0
  2. 2025Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergencecited by 0
  3. 2026Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Progressive Distillation for Fast Sampling of Diffusion Models
  2. 2023Adversarial Diffusion Distillation
  3. 2025Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
  4. 2024Unlocking Dataset Distillation with Diffusion Models
  5. 2026Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations
  6. 2024Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selection
  7. 2022On Distillation of Guided Diffusion Models
  8. 2026DisCa: Accelerating Video Diffusion Transformers with Distillation-Compatible Learnable Feature Caching