# Data-Free Knowledge Transfer: A Survey

arXiv: https://arxiv.org/abs/2112.15278 · HTML: https://mlsanity.com/paper/2112.15278/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 158 of its 213 references that are in the map.
- First posted on arXiv 31 Dec 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 158)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)

## Built on by (most cited first, 30 of 31)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Source-Free Unsupervised Domain Adaptation: A Survey](https://mlsanity.com/paper/2301.00265/) (2022)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning](https://mlsanity.com/paper/2305.13888/) (2023)
- [Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization](https://mlsanity.com/paper/2306.01103/) (2023)
- [DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization](https://mlsanity.com/paper/2408.11071/) (2024)
- [Active Test-Time Adaptation: Theoretical Analyses and An Algorithm](https://mlsanity.com/paper/2404.05094/) (2024)
- [Hard Sample Matters a Lot in Zero-Shot Quantization](https://mlsanity.com/paper/2303.13826/) (2023)
- [Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation](https://mlsanity.com/paper/2410.17606/) (2024)
- [FREE: Faster and Better Data-Free Meta-Learning](https://mlsanity.com/paper/2405.00984/) (2024)
- [Data-Free Adversarial Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2205.03811/) (2022)
- [Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering](https://mlsanity.com/paper/2307.11030/) (2023)
- [Deep Classifier Mimicry without Data Access](https://mlsanity.com/paper/2306.02090/) (2023)
- [Continual Learning From a Stream of APIs](https://mlsanity.com/paper/2309.00023/) (2023)
- [Causal-DFQ: Causality Guided Data-free Network Quantization](https://mlsanity.com/paper/2309.13682/) (2023)
- [DHBE: Data-free Holistic Backdoor Erasing in Deep Neural Networks via Restricted Adversarial Distillation](https://mlsanity.com/paper/2306.08009/) (2023)
- [When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need](https://mlsanity.com/paper/2507.04119/) (2025)
- [Distilling the Undistillable: Learning from a Nasty Teacher](https://mlsanity.com/paper/2210.11728/) (2022)
- [Projected Latent Distillation for Data-Agnostic Consolidation in Distributed Continual Learning](https://mlsanity.com/paper/2303.15888/) (2023)
- [Data-free Knowledge Distillation for Fine-grained Visual Categorization](https://mlsanity.com/paper/2404.12037/) (2024)
- [Encapsulating Knowledge in One Prompt](https://mlsanity.com/paper/2407.11902/) (2024)
- [Training Self-localization Models for Unseen Unfamiliar Places via Teacher-to-Student Data-Free Knowledge Transfer](https://mlsanity.com/paper/2403.10552/) (2024)
- [Condensed Data Expansion Using Model Inversion for Knowledge Distillation](https://mlsanity.com/paper/2408.13850/) (2024)
- [TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models](https://mlsanity.com/paper/2603.24518/) (2026)
- [Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning](https://mlsanity.com/paper/2205.11071/) (2022)
- [Recursive Distillation for Open-Set Distributed Robot Localization](https://mlsanity.com/paper/2312.15897/) (2023)
- [Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective](https://mlsanity.com/paper/2501.07040/) (2025)
- [Task-conditioned Ensemble of Expert Models for Continuous Learning](https://mlsanity.com/paper/2504.08626/) (2025)
- [Bridging Domain Gaps for Fine-Grained Moth Classification Through Expert-Informed Adaptation and Foundation Model Priors](https://mlsanity.com/paper/2508.20089/) (2025)

## Abstract (arXiv)

In the last decade, many deep learning models have been well trained and made a great success in various fields of machine intelligence, especially for computer vision and natural language processing. To better leverage the potential of these well-trained models in intra-domain or cross-domain transfer learning situations, knowledge distillation (KD) and domain adaptation (DA) are proposed and become research highlights. They both aim to transfer useful information from a well-trained model with original training data. However, the original data is not always available in many cases due to privacy, copyright or confidentiality. Recently, the data-free knowledge transfer paradigm has attracted appealing attention as it deals with distilling valuable knowledge from well-trained models without requiring to access to the training data. In particular, it mainly consists of the data-free knowledge distillation (DFKD) and source data-free domain adaptation (SFDA). On the one hand, DFKD aims to transfer the intra-domain knowledge of original data from a cumbersome teacher network to a compact student network for model compression and efficient inference. On the other hand, the goal of SFDA is to reuse the cross-domain knowledge stored in a well-trained source model and adapt it to a target domain. In this paper, we provide a comprehensive survey on data-free knowledge transfer from the perspectives of knowledge distillation and unsupervised domain adaptation, to help readers have a better understanding of the current research status and ideas. Applications and challenges of the two areas are briefly reviewed, respectively. Furthermore, we provide some insights to the subject of future research.