# Large-Scale Generative Data-Free Distillation

arXiv: https://arxiv.org/abs/2012.05578 · HTML: https://mlsanity.com/paper/2012.05578/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 29 of 29)

- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [GradViT: Gradient Inversion of Vision Transformers](https://mlsanity.com/paper/2203.11894/) (2022)
- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2209.10359/) (2022)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation](https://mlsanity.com/paper/2310.00258/) (2023)
- [DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture](https://mlsanity.com/paper/2409.03550/) (2024)
- [Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?](https://mlsanity.com/paper/2305.12954/) (2023)
- [Small Scale Data-Free Knowledge Distillation](https://mlsanity.com/paper/2406.07876/) (2024)
- [Few-Shot Unlearning by Model Inversion](https://mlsanity.com/paper/2205.15567/) (2022)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [Towards Data-Free Domain Generalization](https://mlsanity.com/paper/2110.04545/) (2021)
- [Distilling the Knowledge in Data Pruning](https://mlsanity.com/paper/2403.07854/) (2024)
- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [Sampling to Distill: Knowledge Transfer from Open-World Data](https://mlsanity.com/paper/2307.16601/) (2023)
- [Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy](https://mlsanity.com/paper/2208.13648/) (2022)
- [Representation Consolidation for Training Expert Students](https://mlsanity.com/paper/2107.08039/) (2021)
- [When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need](https://mlsanity.com/paper/2507.04119/) (2025)
- [Robustness-Guided Image Synthesis for Data-Free Quantization](https://mlsanity.com/paper/2310.03661/) (2023)
- [Up to 100× Faster Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.06253/) (2021)
- [A Prototype-Oriented Clustering for Domain Shift with Source Privacy](https://mlsanity.com/paper/2302.03807/) (2023)
- [DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning](https://mlsanity.com/paper/2409.07734/) (2024)
- [Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classification](https://mlsanity.com/paper/2303.07643/) (2023)
- [Conditional Generative Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.15358/) (2021)
- [Tackling Intertwined Data and Device Heterogeneities in Federated Learning with Unlimited Staleness](https://mlsanity.com/paper/2309.13536/) (2023)
- [Seeing the Whole Picture: Distribution-Guided Data-Free Distillation for Semantic Segmentation](https://mlsanity.com/paper/2512.13175/) (2025)

## Abstract (arXiv)

Knowledge distillation is one of the most popular and effective techniques for knowledge transfer, model compression and semi-supervised learning. Most existing distillation approaches require the access to original or augmented training samples. But this can be problematic in practice due to privacy, proprietary and availability concerns. Recent work has put forward some methods to tackle this problem, but they are either highly time-consuming or unable to scale to large datasets. To this end, we propose a new method to train a generative image model by leveraging the intrinsic normalization layers' statistics of the trained teacher network. This enables us to build an ensemble of generators without training data that can efficiently produce substitute inputs for subsequent distillation. The proposed method pushes forward the data-free distillation performance on CIFAR-10 and CIFAR-100 to 95.02% and 77.02% respectively. Furthermore, we are able to scale it to ImageNet dataset, which to the best of our knowledge, has never been done using generative models in a data-free setting.