mlsanityData to 14 Oct 2026

Data-Free Knowledge Distillation for Deep Neural Networks

arXiv 1710.07535 ↗First posted 19 Oct 2017cs.LGcited by 136

Not on the map. Why?

Cited by 66 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.

Key facts. Cited by 136 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 36; 2022: 19; 2023: 27; 2024: 29; 2025: 13; 2026: 12.

136 papers.

Sort:

By area:

  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  2. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  3. 2022A Survey on Generative Diffusion Modelcited by 240
  4. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
  5. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 154
  6. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  7. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  8. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  9. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  10. 2022Factorizing Knowledge in Neural Networkscited by 59
  11. 2022Towards Data-Free Model Stealing in a Hard Label Settingcited by 58
  12. 2021Graph-Free Knowledge Distillation for Graph Neural Networkscited by 46
38 more
  1. 2021Zero-shot Adversarial Quantizationcited by 44
  2. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  3. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 40
  4. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  5. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  6. 2021Visualizing Adapted Knowledge in Domain Transfercited by 32
  7. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  8. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Datacited by 32
  9. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  10. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  11. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  12. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  13. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  14. 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
  15. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 22
  16. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  17. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  18. 2024Small Scale Data-Free Knowledge Distillationcited by 20
  19. 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
  20. 2021AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compressioncited by 17
  21. 2024Text-Enhanced Data-free Approach for Federated Class-Incremental Learningcited by 16
  22. 2024A Review of Pseudo-Labeling for Computer Visioncited by 16
  23. 2021Generative Zero-shot Network Quantizationcited by 15
  24. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  25. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesiscited by 14
  26. 2022Black-box Few-shot Knowledge Distillationcited by 14
  27. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 13
  28. 2025Does Training with Synthetic Data Truly Protect Privacy?cited by 12
  29. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11
  30. 2021Towards Data-Free Domain Generalizationcited by 11
  31. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shiftscited by 11
  32. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  33. 2021Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regressioncited by 10
  34. 2021Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillationcited by 10
  35. 2023Architecture, Dataset and Model-Scale Agnostic Data-free Meta-Learningcited by 10
  36. 2021Dual Discriminator Adversarial Distillation for Data-free Model Compressioncited by 9
  37. 2021Thief, Beware of What Get You There: Towards Understanding Model Extraction Attackcited by 9
  38. 2023SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Datacited by 9

50 most cited of 136 All 136 (JSON)