mlsanityData to 14 Oct 2026

Zero-shot Knowledge Transfer via Adversarial Belief Matching

arXiv 1905.09768 ↗First posted 23 May 2019cs.LG stat.MLcited by 129

Not on the map. Why?

Cited by 59 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel method which trains a student to match the predictions of its teacher without using any data or metadata. We achieve this by training an adversarial generator to search for images on which the student poorly matches the teacher, and then using them to train the student. Our resulting student closely approximates its teacher for simple datasets like SVHN, and on CIFAR10 we improve on the state-of-the-art for few-shot distillation (with 100 images per class), despite using no data. Finally, we also propose a metric to quantify the degree of belief matching between teacher and student in the vicinity of decision boundaries, and observe a significantly higher match between our zero-shot student and the teacher, than between a student distilled with real data and the teacher. Code available at: https://github.com/polo5/ZeroShotKnowledgeTransfer

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 22; 2023: 29; 2024: 23; 2025: 17; 2026: 6.

129 papers.

Sort:

By area:

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  2. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  3. 2022Large Language Models Are Reasoning Teacherscited by 329
  4. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  5. 2022A Survey of Machine Unlearningcited by 312
  6. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
  7. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
  8. 2022Zero-Shot Machine Unlearningcited by 133
  9. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 116
  10. 2023A Comprehensive Survey on Source-free Domain Adaptationcited by 104
  11. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replaycited by 72
  12. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
38 more
  1. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  2. 2021Graph-Free Knowledge Distillation for Graph Neural Networkscited by 46
  3. 2022A Survey on Heterogeneous Federated Learningcited by 42
  4. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 40
  5. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  6. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
  7. 2023Lion: Adversarial Distillation of Proprietary Large Language Modelscited by 34
  8. 2021Visualizing Adapted Knowledge in Domain Transfercited by 32
  9. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Datacited by 32
  10. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Datacited by 32
  11. 2022Deep Regression Unlearningcited by 32
  12. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  13. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  14. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  15. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  16. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  17. 2023Federated Domain Generalization: A Surveycited by 25
  18. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  19. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  20. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 22
  21. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  22. 2024Small Scale Data-Free Knowledge Distillationcited by 20
  23. 2021AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compressioncited by 17
  24. 2022IDEAL: Query-Efficient Data-Free Learning from Black-box Modelscited by 16
  25. 2024Text-Enhanced Data-free Approach for Federated Class-Incremental Learningcited by 16
  26. 2025Data-free Knowledge Distillation with Diffusion Modelscited by 16
  27. 2021Generative Zero-shot Network Quantizationcited by 15
  28. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  29. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesiscited by 14
  30. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 13
  31. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11
  32. 2021Towards Data-Free Domain Generalizationcited by 11
  33. 2021FedRAD: Federated Robust Adaptive Distillationcited by 11
  34. 2023Generative Adversarial Networks Unlearningcited by 11
  35. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shiftscited by 11
  36. 2021Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillationcited by 10
  37. 2023Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attackscited by 10
  38. 2021Dual Discriminator Adversarial Distillation for Data-free Model Compressioncited by 9

50 most cited of 129 All 129 (JSON)