mlsanityData to 14 Oct 2026

Data-Free Knowledge Transfer: A Survey

arXiv 2112.15278 ↗First posted 31 Dec 2021cs.LG cs.CVcited by 31

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

In the last decade, many deep learning models have been well trained and made a great success in various fields of machine intelligence, especially for computer vision and natural language processing. To better leverage the potential of these well-trained models in intra-domain or cross-domain transfer learning situations, knowledge distillation (KD) and domain adaptation (DA) are proposed and become research highlights. They both aim to transfer useful information from a well-trained model with original training data. However, the original data is not always available in many cases due to privacy, copyright or confidentiality. Recently, the data-free knowledge transfer paradigm has attracted appealing attention as it deals with distilling valuable knowledge from well-trained models without requiring to access to the training data. In particular, it mainly consists of the data-free knowledge distillation (DFKD) and source data-free domain adaptation (SFDA). On the one hand, DFKD aims to transfer the intra-domain knowledge of original data from a cumbersome teacher network to a compact student network for model compression and efficient inference. On the other hand, the goal of SFDA is to reuse the cross-domain knowledge stored in a well-trained source model and adapt it to a target domain. In this paper, we provide a comprehensive survey on data-free knowledge transfer from the perspectives of knowledge distillation and unsupervised domain adaptation, to help readers have a better understanding of the current research status and ideas. Applications and challenges of the two areas are briefly reviewed, respectively. Furthermore, we provide some insights to the subject of future research.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 158 of its 213 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

158 of 213 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data27 Oct
  2. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data11 Aug
  3. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model7 Jun
  4. 2021Contrastive Model Inversion for Data-Free Knowledge Distillation18 May
  5. 2021Dual Discriminator Adversarial Distillation for Data-free Model Compression12 Apr
  6. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation6 Mar

Other references · 152

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  11. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
38 more
  1. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  9. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,082
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  11. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  12. 2019Deep Leakage from Gradientscited by 1,021
  13. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  14. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  15. 2019Relational Knowledge Distillationcited by 889
  16. 2020Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptationcited by 848
  17. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  18. 2018Deep Visual Domain Adaptation: A Surveycited by 635
  19. 2017Deep Mutual Learningcited by 628
  20. 2018ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentationcited by 623
  21. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  22. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  23. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  24. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  25. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  26. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  27. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  28. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  29. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  30. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  31. 2019Bidirectional Learning for Domain Adaptation of Semantic Segmentationcited by 308
  32. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 306
  33. 2018Model compression via distillation and quantizationcited by 289
  34. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  35. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  36. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  37. 2019Learning to Discover Novel Visual Categories via Deep Transfer Clusteringcited by 225
  38. 2020Universal Source-Free Domain Adaptationcited by 220

50 most cited of 152 All 152 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 12; 2024: 8; 2025: 4; 2026: 2.

31 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  2. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  3. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 80
  4. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 65
  5. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  6. 2023Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalizationcited by 29
  7. 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 22
  8. 2024Active Test-Time Adaptation: Theoretical Analyses and An Algorithmcited by 18
  9. 2023Hard Sample Matters a Lot in Zero-Shot Quantizationcited by 15
  10. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 13
  11. 2024FREE: Faster and Better Data-Free Meta-Learningcited by 8
  12. 2022Data-Free Adversarial Knowledge Distillation for Graph Neural Networkscited by 7
19 more
  1. 2023Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clusteringcited by 7
  2. 2023Deep Classifier Mimicry without Data Accesscited by 6
  3. 2023Continual Learning From a Stream of APIscited by 6
  4. 2023Causal-DFQ: Causality Guided Data-free Network Quantizationcited by 6
  5. 2023DHBE: Data-free Holistic Backdoor Erasing in Deep Neural Networks via Restricted Adversarial Distillationcited by 5
  6. 2025When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Needcited by 5
  7. 2022Distilling the Undistillable: Learning from a Nasty Teachercited by 4
  8. 2023Projected Latent Distillation for Data-Agnostic Consolidation in Distributed Continual Learningcited by 4
  9. 2024Data-free Knowledge Distillation for Fine-grained Visual Categorizationcited by 4
  10. 2024Encapsulating Knowledge in One Promptcited by 3
  11. 2024Training Self-localization Models for Unseen Unfamiliar Places via Teacher-to-Student Data-Free Knowledge Transfercited by 1
  12. 2024Condensed Data Expansion Using Model Inversion for Knowledge Distillationcited by 1
  13. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  14. 2022Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learningcited by 0
  15. 2023Recursive Distillation for Open-Set Distributed Robot Localizationcited by 0
  16. 2025Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspectivecited by 0
  17. 2025Task-conditioned Ensemble of Expert Models for Continuous Learningcited by 0
  18. 2025Bridging Domain Gaps for Fine-Grained Moth Classification Through Expert-Informed Adaptation and Foundation Model Priorscited by 0
  19. 2026UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
  2. 2024Direct Distillation between Different Domains
  3. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts
  4. 2023Sampling to Distill: Knowledge Transfer from Open-World Data
  5. 2023Black-box Source-free Domain Adaptation via Two-stage Knowledge Distillation
  6. 2024AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation
  7. 2024Small Scale Data-Free Knowledge Distillation
  8. 2025Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation