# Accelerating Diffusion Models with One-to-Many Knowledge Distillation

arXiv: https://arxiv.org/abs/2410.04191 · HTML: https://mlsanity.com/paper/2410.04191/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 5 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed](https://mlsanity.com/paper/2101.02388/) (2021)
- [Denoising Diffusion Models for Plug-and-Play Image Restoration](https://mlsanity.com/paper/2305.08995/) (2023)
- [Post-training Quantization on Diffusion Models](https://mlsanity.com/paper/2211.15736/) (2022)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [Accelerating Diffusion Sampling with Classifier-based Feature Distillation](https://mlsanity.com/paper/2211.12039/) (2022)

## Built on by (most cited first, 9 of 9)

- [RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectories](https://mlsanity.com/paper/2503.07699/) (2025)
- [Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformers](https://mlsanity.com/paper/2507.08422/) (2025)
- [Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas](https://mlsanity.com/paper/2506.03275/) (2025)
- [Denoising as Path Planning: Training-Free Acceleration of Diffusion Models with DPCache](https://mlsanity.com/paper/2602.22654/) (2026)
- [DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.16968/) (2026)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models](https://mlsanity.com/paper/2606.00798/) (2026)
- [2ndMatch: Finetuning Pruned Diffusion Models via Second-Order Jacobian Matching](https://mlsanity.com/paper/2506.05398/) (2025)
- [LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition](https://mlsanity.com/paper/2509.15342/) (2025)

## Abstract (arXiv)

Significant advancements in image generation have been made with diffusion models. Nevertheless, when contrasted with previous generative models, diffusion models face substantial computational overhead, leading to failure in real-time generation. Recent approaches have aimed to accelerate diffusion models by reducing the number of sampling steps through improved sampling techniques or step distillation. However, the methods to diminish the computational cost for each timestep remain a relatively unexplored area. Observing the fact that diffusion models exhibit varying input distributions and feature distributions at different timesteps, we introduce one-to-many knowledge distillation (O2MKD), which distills a single teacher diffusion model into multiple student diffusion models, where each student diffusion model is trained to learn the teacher's knowledge for a subset of continuous timesteps. Experiments on CIFAR10, LSUN Church, CelebA-HQ with DDPM and COCO30K with Stable Diffusion show that O2MKD can be applied to previous knowledge distillation and fast sampling methods to achieve significant acceleration. Codes will be released in Github.