# Data-Free Learning of Student Networks

arXiv: https://arxiv.org/abs/1904.01186 · HTML: https://mlsanity.com/paper/1904.01186/

## Key facts

- Cited by 204 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 30 of 204)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [A Comprehensive Survey on Source-free Domain Adaptation](https://mlsanity.com/paper/2302.11803/) (2023)
- [Domain Impression: A Source Data Free Domain Adaptation Method](https://mlsanity.com/paper/2102.09003/) (2021)
- [Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay](https://mlsanity.com/paper/2207.11213/) (2022)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers](https://mlsanity.com/paper/2204.12997/) (2022)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [GradViT: Gradient Inversion of Vision Transformers](https://mlsanity.com/paper/2203.11894/) (2022)
- [Graph-Free Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2105.07519/) (2021)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [Deep Learning for HDR Imaging: State-of-the-Art and Future Trends](https://mlsanity.com/paper/2110.10394/) (2021)
- [Zero-shot Adversarial Quantization](https://mlsanity.com/paper/2103.15263/) (2021)
- [A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks](https://mlsanity.com/paper/2311.07784/) (2023)
- [Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples](https://mlsanity.com/paper/2111.02625/) (2021)
- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [Undistillable: Making A Nasty Teacher That CANNOT teach students](https://mlsanity.com/paper/2105.07381/) (2021)
- [Evaluations of Machine Learning Privacy Defenses are Misleading](https://mlsanity.com/paper/2404.17399/) (2024)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)
- [Visualizing Adapted Knowledge in Domain Transfer](https://mlsanity.com/paper/2104.10602/) (2021)
- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)

## Abstract (arXiv)

Learning portable neural networks is very essential for computer vision for the purpose that pre-trained heavy deep models can be well applied on edge devices such as mobile phones and micro sensors. Most existing deep neural network compression and speed-up methods are very effective for training compact deep models, when we can directly access the training dataset. However, training data for the given deep network are often unavailable due to some practice problems (e.g. privacy, legal issue, and transmission), and the architecture of the given network are also unknown except some interfaces. To this end, we propose a novel framework for training efficient deep neural networks by exploiting generative adversarial networks (GANs). To be specific, the pre-trained teacher networks are regarded as a fixed discriminator and the generator is utilized for derivating training samples which can obtain the maximum response on the discriminator. Then, an efficient network with smaller model size and computational complexity is trained using the generated data and the teacher network, simultaneously. Efficient student networks learned using the proposed Data-Free Learning (DAFL) method achieve 92.22% and 74.47% accuracies using ResNet-18 without any training data on the CIFAR-10 and CIFAR-100 datasets, respectively. Meanwhile, our student network obtains an 80.56% accuracy on the CelebA benchmark.