# Plug-and-Play Diffusion Distillation

arXiv: https://arxiv.org/abs/2406.01954 · HTML: https://mlsanity.com/paper/2406.01954/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 28 of its 30 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Gotta Go Fast When Generating Data with Score-Based Models](https://mlsanity.com/paper/2105.14080/) (2021)
- [LCM-LoRA: A Universal Stable-Diffusion Acceleration Module](https://mlsanity.com/paper/2311.05556/) (2023)
- [gDDIM: Generalized denoising diffusion implicit models](https://mlsanity.com/paper/2206.05564/) (2022)
- [Patched Denoising Diffusion Models For High-Resolution Image Synthesis](https://mlsanity.com/paper/2308.01316/) (2023)
- [Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?](https://mlsanity.com/paper/2305.09847/) (2023)

## Built on by (most cited first, 16 of 16)

- [FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality](https://mlsanity.com/paper/2410.19355/) (2024)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [TCFG: Tangential Damping Classifier-free Guidance](https://mlsanity.com/paper/2503.18137/) (2025)
- [Learnable Sparsity for Vision Generative Models](https://mlsanity.com/paper/2412.02852/) (2024)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Efficient Distillation of Classifier-Free Guidance using Adapters](https://mlsanity.com/paper/2503.07274/) (2025)
- [HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models](https://mlsanity.com/paper/2506.19072/) (2025)
- [Diffusion Sampling Correction via Approximately 10 Parameters](https://mlsanity.com/paper/2411.06503/) (2024)
- [Momentum Guidance: Plug-and-Play Guidance for Flow Models](https://mlsanity.com/paper/2602.20360/) (2026)
- [Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation](https://mlsanity.com/paper/2511.18281/) (2025)
- [FlexControl: Computation-Aware ControlNet with Differentiable Router for Text-to-Image Generation](https://mlsanity.com/paper/2502.10451/) (2025)
- [Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications](https://mlsanity.com/paper/2511.08416/) (2025)
- [Real-time cardiac cine MRI -- A comparison of a diffusion probabilistic model with alternative state-of-the-art image reconstruction techniques for undersampled spiral acquisitions](https://mlsanity.com/paper/2410.04843/) (2024)
- [FlowBlending: Stage-Aware Multi-Model Sampling for Fast and High-Fidelity Video Generation](https://mlsanity.com/paper/2512.24724/) (2025)
- [Weak-to-Strong Knowledge Distillation Accelerates Visual Learning](https://mlsanity.com/paper/2604.15451/) (2026)
- [LongLive-Plug: Once-for-All Distillation for Video Generation](https://mlsanity.com/paper/2609.38154/) (2026)

## Abstract (arXiv)

Diffusion models have shown tremendous results in image generation. However, due to the iterative nature of the diffusion process and its reliance on classifier-free guidance, inference times are slow. In this paper, we propose a new distillation approach for guided diffusion models in which an external lightweight guide model is trained while the original text-to-image model remains frozen. We show that our method reduces the inference computation of classifier-free guided latent-space diffusion models by almost half, and only requires 1% trainable parameters of the base model. Furthermore, once trained, our guide model can be applied to various fine-tuned, domain-specific versions of the base diffusion model without the need for additional training: this "plug-and-play" functionality drastically improves inference computation while maintaining the visual fidelity of generated images. Empirically, we show that our approach is able to produce visually appealing results and achieve a comparable FID score to the teacher with as few as 8 to 16 steps.