mlsanityData to 14 Oct 2026

Data-free Knowledge Distillation with Diffusion Models

arXiv 2504.00870 ↗First posted 1 Apr 2025cs.CVcited by 16

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recently Data-Free Knowledge Distillation (DFKD) has garnered attention and can transfer knowledge from a teacher neural network to a student neural network without requiring any access to training data. Although diffusion models are adept at synthesizing high-fidelity photorealistic images across various domains, existing methods cannot be easiliy implemented to DFKD. To bridge that gap, this paper proposes a novel approach based on diffusion models, DiffDFKD. Specifically, DiffDFKD involves targeted optimizations in two key areas. Firstly, DiffDFKD utilizes valuable information from teacher models to guide the pre-trained diffusion models' data synthesis, generating datasets that mirror the training data distribution and effectively bridge domain gaps. Secondly, to reduce computational burdens, DiffDFKD introduces Latent CutMix Augmentation, an efficient technique, to enhance the diversity of diffusion model-generated images for DFKD while preserving key attributes for effective knowledge transfer. Extensive experiments validate the efficacy of DiffDFKD, yielding state-of-the-art results exceeding existing DFKD approaches. We release our code at https://github.com/xhqi0109/DiffDFKD.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 21 of its 29 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

21 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?22 May
  2. 2022Scalable Diffusion Models with Transformers19 Dec
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec

Other references · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2016Wide Residual Networkscited by 3,070
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  8. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  9. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  10. 2019Data-Free Learning of Student Networkscited by 204
  11. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  12. 2022A Comprehensive Survey of Image Augmentation Techniques for Deep Learningcited by 97
6 more
  1. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  2. 2024Towards Realistic Data Generation for Real-World Super-Resolutioncited by 56
  3. 2024Lightweight Adaptive Feature De-drifting for Compressed Image Classificationcited by 46
  4. 2023Federated Generative Learning with Foundation Modelscited by 23
  5. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  6. 2024Small Scale Data-Free Knowledge Distillationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025RobustGS: Unified Boosting of Feedforward 3D Gaussian Splatting under Low-Quality Conditionscited by 13
  2. 2026Depth-Synergized Mamba Meets Memory Experts for All-Day Image Reflection Separationcited by 5
  3. 2025Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignmentcited by 4
  4. 2025Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarkscited by 3
  5. 2025SSCM: A Spatial-Semantic Consistent Model for Multi-Contrast MRI Super-Resolutioncited by 2
  6. 2026Vanishing Watermarks: Diffusion-Based Image Editing Undermines Robust Invisible Watermarkingcited by 2
  7. 2026Editing on the Generative Manifold: A Theoretical and Empirical Study of General Diffusion-Based Image Editing Trade-offscited by 1
  8. 2025SemaMIL: Semantic-Aware Multiple Instance Learning with Retrieval-Guided State Space Modeling for Whole Slide Imagescited by 0
  9. 2025DeRainMamba: A Frequency-Aware State Space Model with Detail Enhancement for Image Derainingcited by 0
  10. 2025ReDiF: Reinforced Distillation for Few Step Diffusioncited by 0
  11. 2026When Denoising Becomes Unsigning: Theoretical and Empirical Analysis of Watermark Fragility Under Diffusion-Based Image Editingcited by 0
  12. 2026Editing Away the Evidence: Diffusion-Based Image Manipulation and the Failure Modes of Robust Watermarkingcited by 0
4 more
  1. 2026Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoderscited by 0
  2. 2026ReconMIL: Synergizing Latent Space Reconstruction with Bi-Stream Mamba for Whole Slide Image Analysiscited by 0
  3. 2026Gradient Transformer: Learning to Generate Updates for LLMscited by 0
  4. 2026UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation
  2. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture
  3. 2024Relational Diffusion Distillation for Efficient Image Generation
  4. 2024Accelerating Diffusion Models with One-to-Many Knowledge Distillation
  5. 2024Plug-and-Play Diffusion Distillation
  6. 2023Knowledge Diffusion for Distillation
  7. 2025Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression
  8. 2024SFDDM: Single-fold Distillation for Diffusion models