mlsanityData to 7 Oct 2026

Zero-Shot Knowledge Distillation in Deep Networks

arXiv 1905.08114 ↗First posted 20 May 2019cs.LG cs.CV stat.MLcited by 119

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which the Teacher has been trained may not always be feasible if the dataset is very large or it poses privacy or safety concerns (e.g., bio-metric or medical data). Hence, in this paper, we propose a novel data-free method to train the Student from the Teacher. Without even using any meta-data, we synthesize the Data Impressions from the complex Teacher model and utilize these as surrogates for the original training data samples to transfer its learning to Student via knowledge distillation. We, therefore, dub our method "Zero-Shot Knowledge Distillation" and demonstrate that our framework results in competitive generalization performance as achieved by distillation using the actual training data samples on multiple benchmark datasets.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 20; 2023: 27; 2024: 20; 2025: 5; 2026: 8.

119 papers.

Sort:

By area:

  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  2. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  3. 2022Large Language Models Are Reasoning Teacherscited by 318
  4. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  5. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
  6. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  7. 2021Adversarial Examples Make Strong Poisonscited by 115
  8. 2023A Comprehensive Survey on Source-free Domain Adaptationcited by 104
  9. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  10. 2021Domain Impression: A Source Data Free Domain Adaptation Methodcited by 91
  11. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  12. 2022Towards Data-Free Model Stealing in a Hard Label Settingcited by 57
38 more
  1. 2021Graph-Free Knowledge Distillation for Graph Neural Networkscited by 46
  2. 2024Lightweight Adaptive Feature De-drifting for Compressed Image Classificationcited by 46
  3. 2021Zero-shot Adversarial Quantizationcited by 44
  4. 2021DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memoriescited by 42
  5. 2022A Survey on Heterogeneous Federated Learningcited by 42
  6. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 39
  7. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  8. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
  9. 2021Visualizing Adapted Knowledge in Domain Transfercited by 32
  10. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Datacited by 32
  11. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  12. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  13. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  14. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  15. 2024FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learningcited by 27
  16. 2023Federated Domain Generalization: A Surveycited by 25
  17. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
  18. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  19. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 21
  20. 2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
  21. 2024Small Scale Data-Free Knowledge Distillationcited by 20
  22. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  23. 2021AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compressioncited by 17
  24. 2021Generative Zero-shot Network Quantizationcited by 15
  25. 2021PrivateSNN: Privacy-Preserving Spiking Neural Networkscited by 15
  26. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  27. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesiscited by 14
  28. 2022Black-box Few-shot Knowledge Distillationcited by 14
  29. 2021Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledgecited by 12
  30. 2022Generalized Knowledge Distillation via Relationship Matchingcited by 12
  31. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 12
  32. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11
  33. 2021LANA: Latency Aware Network Accelerationcited by 11
  34. 2021Towards Data-Free Domain Generalizationcited by 11
  35. 2021FedRAD: Federated Robust Adaptive Distillationcited by 11
  36. 2021Ex-Model: Continual Learning from a Stream of Trained Modelscited by 11
  37. 2024Distilling the Knowledge in Data Pruningcited by 11
  38. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10

50 most cited of 119 All 119 (JSON)