# DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture

arXiv: https://arxiv.org/abs/2409.03550 · HTML: https://mlsanity.com/paper/2409.03550/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 68 of its 83 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 68)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)

## Built on by (most cited first, 22 of 22)

- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration](https://mlsanity.com/paper/2504.15159/) (2025)
- [Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression](https://mlsanity.com/paper/2504.02011/) (2025)
- [Training Data Provenance Verification: Did Your Model Use Synthetic Data from My Generative Model for Training?](https://mlsanity.com/paper/2503.09122/) (2025)
- [Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models](https://mlsanity.com/paper/2601.20687/) (2026)
- [TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models](https://mlsanity.com/paper/2603.17828/) (2026)
- [Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models](https://mlsanity.com/paper/2409.12512/) (2024)
- [ParseFixer: An Agentic Framework for Document Parsing via Selective Multimodal Correction](https://mlsanity.com/paper/2606.11977/) (2026)
- [Upcycling Text-to-Image Diffusion Models for Multi-Task Capabilities](https://mlsanity.com/paper/2503.11905/) (2025)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Constrained Diffusion Models for Synthesizing Representative Power Flow Datasets](https://mlsanity.com/paper/2506.11281/) (2025)
- [ViSAGE @ NTIRE 2026 Challenge on Video Saliency Prediction](https://mlsanity.com/paper/2604.08613/) (2026)
- [ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement](https://mlsanity.com/paper/2606.10640/) (2026)
- [TINA+: Probing Residual Visual Knowledge in Unlearned Diffusion Models via Diffusion-Consistent Text-Free Inversion](https://mlsanity.com/paper/2608.17747/) (2026)
- [KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generation](https://mlsanity.com/paper/2505.06995/) (2025)
- [Faster and Better: Reinforced Collaborative Distillation and Self-Learning for Infrared-Visible Image Fusion](https://mlsanity.com/paper/2509.02424/) (2025)
- [ReDiF: Reinforced Distillation for Few Step Diffusion](https://mlsanity.com/paper/2512.22802/) (2025)
- [B-DENSE: Branching For Dense Ensemble Network Supervision Efficiency](https://mlsanity.com/paper/2602.15971/) (2026)
- [On the Memorization of Consistency Distillation for Diffusion Models](https://mlsanity.com/paper/2604.23552/) (2026)
- [LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models](https://mlsanity.com/paper/2605.19729/) (2026)
- [ARIA: Adaptive Region-Based Importance Allocation for Conditional Diffusion Distillation](https://mlsanity.com/paper/2606.23898/) (2026)
- [TACD: Distilling Efficient Text-to-Motion Models via Terminal Amplification Control](https://mlsanity.com/paper/2610.02867/) (2026)

## Abstract (arXiv)

Diffusion models (DMs) have demonstrated exceptional generative capabilities across various domains, including image, video, and so on. A key factor contributing to their effectiveness is the high quantity and quality of data used during training. However, mainstream DMs now consume increasingly large amounts of data. For example, training a Stable Diffusion model requires billions of image-text pairs. This enormous data requirement poses significant challenges for training large DMs due to high data acquisition costs and storage expenses. To alleviate this data burden, we propose a novel scenario: using existing DMs as data sources to train new DMs with any architecture. We refer to this scenario as Data-Free Knowledge Distillation for Diffusion Models (DKDM), where the generative ability of DMs is transferred to new ones in a data-free manner. To tackle this challenge, we make two main contributions. First, we introduce a DKDM objective that enables the training of new DMs via distillation, without requiring access to the data. Second, we develop a dynamic iterative distillation method that efficiently extracts time-domain knowledge from existing DMs, enabling direct retrieval of training data without the need for a prolonged generative process. To the best of our knowledge, we are the first to explore this scenario. Experimental results demonstrate that our data-free approach not only achieves competitive generative performance but also, in some instances, outperforms models trained with the entire dataset.