# Accelerating Diffusion Sampling with Classifier-based Feature Distillation

arXiv: https://arxiv.org/abs/2211.12039 · HTML: https://mlsanity.com/paper/2211.12039/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed](https://mlsanity.com/paper/2101.02388/) (2021)
- [On Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.00132/) (2021)
- [Channel-wise Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/2011.13256/) (2020)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [Generative Adversarial Networks (GANs Survey): Challenges, Solutions, and Future Directions](https://mlsanity.com/paper/2005.00065/) (2020)
- [BIGRoC: Boosting Image Generation via a Robust Classifier](https://mlsanity.com/paper/2108.03702/) (2021)

## Built on by (most cited first, 15 of 15)

- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [A Comprehensive Survey on Knowledge Distillation of Diffusion Models](https://mlsanity.com/paper/2304.04262/) (2023)
- [Simple and Fast Distillation of Diffusion Models](https://mlsanity.com/paper/2409.19681/) (2024)
- [Towards a mathematical theory for consistency training in diffusion models](https://mlsanity.com/paper/2402.07802/) (2024)
- [Catch-Up Distillation: You Only Need to Train Once for Accelerating Sampling](https://mlsanity.com/paper/2305.10769/) (2023)
- [Knowledge Distillation with Refined Logits](https://mlsanity.com/paper/2408.07703/) (2024)
- [Accelerating Diffusion Models with One-to-Many Knowledge Distillation](https://mlsanity.com/paper/2410.04191/) (2024)
- [FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training](https://mlsanity.com/paper/2506.10035/) (2025)
- [Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models](https://mlsanity.com/paper/2407.00626/) (2024)
- [Theoretical research on generative diffusion models: an overview](https://mlsanity.com/paper/2404.09016/) (2024)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation](https://mlsanity.com/paper/2408.15991/) (2024)
- [Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence](https://mlsanity.com/paper/2503.06685/) (2025)
- [Knowledge Translation: A New Pathway for Model Compression](https://mlsanity.com/paper/2401.05772/) (2024)
- [Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model](https://mlsanity.com/paper/2602.02107/) (2026)

## Abstract (arXiv)

Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of N-step teacher sampler with N/2-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed Classifier-based Feature Distillation (CFD). Instead of aligning output images, we distill teacher's sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance. We also introduce a dataset-oriented loss to further optimize the model. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling. Code is provided at urlhttps://github.com/zju-SWJ/RCFD.