mlsanityData to 7 Oct 2026

Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?

arXiv 2305.12954 ↗First posted 22 May 2023cs.CVcited by 21

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images. Given their huge success, it is still unclear whether synthetic images are applicable for knowledge distillation when real images are unavailable. In this paper, we extensively study whether and how synthetic images produced from state-of-the-art diffusion models can be used for knowledge distillation without access to real images, and obtain three key conclusions: (1) synthetic data from diffusion models can easily lead to state-of-the-art performance among existing synthesis-based distillation methods, (2) low-fidelity synthetic images are better teaching materials, and (3) relatively weak classifiers are better teachers. Code is available at https://github.com/zhengli97/DM-KD.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 62 of its 69 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

62 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  6. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  7. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  9. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  5. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  6. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  9. 2019Relational Knowledge Distillationcited by 873
  10. 2019Contrastive Representation Distillationcited by 734
  11. 2017Deep Mutual Learningcited by 621
  12. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  13. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  14. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  15. 2022Decoupled Knowledge Distillationcited by 380
  16. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  17. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  18. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  19. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  20. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  21. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  22. 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Modelscited by 215
  23. 2019Data-Free Learning of Student Networkscited by 203
  24. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  25. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  26. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  27. 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 106
  28. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 79
  29. 2018Fast Human Pose Estimationcited by 68
  30. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  31. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  32. 2019The Knowledge Within: Methods for Data-Free Model Compressioncited by 54
  33. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
  34. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  35. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  36. 2020Large-Scale Generative Data-Free Distillationcited by 29
  37. 2022Switchable Online Knowledge Distillationcited by 29
  38. 2022Improved Feature Distillation via Projector Ensemblecited by 27

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 6; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023Diffusion Model as Representation Learnercited by 95
  2. 2024DiffuseMix: Label-Preserving Data Augmentation with Diffusion Modelscited by 78
  3. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  4. 2023DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Controlcited by 44
  5. 2023Federated Generative Learning with Foundation Modelscited by 23
  6. 2023Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Modelscited by 18
  7. 2024ODGEN: Domain-specific Object Detection Data Generation with Diffusion Modelscited by 17
  8. 2025Data-free Knowledge Distillation with Diffusion Modelscited by 16
  9. 2024GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editingcited by 14
  10. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 8
  11. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 8
  12. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
9 more
  1. 2024Rethinking cluster-conditioned diffusion models for label-free image synthesiscited by 5
  2. 2024Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Datacited by 4
  3. 2023ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schr"odinger Bridgecited by 3
  4. 2025D-Feat Occlusions: Diffusion Features for Robustness to Partial Visual Occlusions in Object Recognitioncited by 2
  5. 2024Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentationcited by 1
  6. 2025PRISM: Precision-Recall Informed Data-Free Knowledge Distillation via Generative Diffusioncited by 1
  7. 2025Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learningcited by 1
  8. 2025When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Classcited by 0
  9. 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data
  2. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture
  3. 2025A Survey on Pre-Trained Diffusion Model Distillations
  4. 2022Evaluating a Synthetic Image Dataset Generated with Stable Diffusion
  5. 2024SFDDM: Single-fold Distillation for Diffusion models
  6. 2024Relational Diffusion Distillation for Efficient Image Generation
  7. 2024Accelerating Diffusion Models with One-to-Many Knowledge Distillation
  8. 2026When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators