mlsanityData to 14 Oct 2026

Data-Free Adversarial Distillation

arXiv 1912.11006 ↗First posted 23 Dec 2019cs.LG cs.CV stat.MLcited by 90

Not on the map. Why?

Cited by 42 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer. However, almost all existing KD algorithms are data-driven, i.e., relying on a large amount of original training data or alternative data, which is usually unavailable in real-world scenarios. In this paper, we devote ourselves to this challenging problem and propose a novel adversarial distillation mechanism to craft a compact student model without any real-world data. We introduce a model discrepancy to quantificationally measure the difference between student and teacher models and construct an optimizable upper bound. In our work, the student and the teacher jointly act the role of the discriminator to reduce this discrepancy, when a generator adversarially produces some "hard samples" to enlarge it. Extensive experiments demonstrate that the proposed data-free method yields comparable performance to existing data-driven methods. More strikingly, our approach can be directly extended to semantic segmentation, which is more complicated than classification, and our approach achieves state-of-the-art results. Code and pretrained models are available at https://github.com/VainF/Data-Free-Adversarial-Distillation.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 21; 2022: 16; 2023: 20; 2024: 17; 2025: 11; 2026: 5.

90 papers.

Sort:

By area:

  1. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  2. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
  3. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 116
  4. 2021Distilling Object Detectors via Decoupled Featurescited by 106
  5. 2022Synthesizing Informative Training Samples with GANcited by 77
  6. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  7. 2022Towards Data-Free Model Stealing in a Hard Label Settingcited by 58
  8. 2021Defending against Model Stealing via Verifying Embedded External Featurescited by 46
  9. 2021Zero-shot Adversarial Quantizationcited by 44
  10. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 40
  11. 2021MEGEX: Data-Free Model Extraction Attack against Gradient-Based Explainable AIcited by 35
  12. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replaycited by 35
38 more
  1. 2023Lion: Adversarial Distillation of Proprietary Large Language Modelscited by 34
  2. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Datacited by 32
  3. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Datacited by 32
  4. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  5. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  6. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  7. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  8. 2021Source-free unsupervised domain adaptation for cross-modality abdominal multi-organ segmentationcited by 23
  9. 2022Spot-adaptive Knowledge Distillationcited by 22
  10. 2022MOVE: Effective and Harmless Ownership Verification via Embedded External Featurescited by 21
  11. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  12. 2024Small Scale Data-Free Knowledge Distillationcited by 20
  13. 2021AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compressioncited by 17
  14. 2023Dual Student Networks for Data-Free Model Stealingcited by 17
  15. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  16. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11
  17. 2021Dual Discriminator Adversarial Distillation for Data-free Model Compressioncited by 9
  18. 2023Dataset Distillation via Adversarial Prediction Matchingcited by 9
  19. 2023Explicit and Implicit Knowledge Distillation via Unlabeled Datacited by 8
  20. 2023Distribution Shift Matters for Knowledge Distillation with Webly Collected Imagescited by 8
  21. 2023Sampling to Distill: Knowledge Transfer from Open-World Datacited by 8
  22. 2021Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolationcited by 7
  23. 2022Data-Free Adversarial Knowledge Distillation for Graph Neural Networkscited by 7
  24. 2022Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategycited by 7
  25. 2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
  26. 2023NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistencycited by 7
  27. 2025Forget the Data and Fine-Tuning! Just Fold the Network to Compresscited by 7
  28. 2021Training Generative Adversarial Networks in One Stagecited by 6
  29. 2023Continual Learning From a Stream of APIscited by 6
  30. 2023Causal-DFQ: Causality Guided Data-free Network Quantizationcited by 6
  31. 2023Fake It Till Make It: Federated Learning with Consensus-Oriented Generationcited by 6
  32. 2023Federated Learning via Input-Output Collaborative Distillationcited by 6
  33. 2024Relation-Guided Adversarial Learning for Data-free Knowledge Transfercited by 6
  34. 2021Representation Consolidation for Training Expert Studentscited by 5
  35. 2023DHBE: Data-free Holistic Backdoor Erasing in Deep Neural Networks via Restricted Adversarial Distillationcited by 5
  36. 2024Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillationcited by 5
  37. 2024Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversificationcited by 5
  38. 2025Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignmentcited by 5

50 most cited of 90 All 90 (JSON)