# Robustness and Diversity Seeking Data-Free Knowledge Distillation

arXiv: https://arxiv.org/abs/2011.03749 · HTML: https://mlsanity.com/paper/2011.03749/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2209.10359/) (2022)
- [Careful What You Wish For: on the Extraction of Adversarially Trained Models](https://mlsanity.com/paper/2207.10561/) (2022)
- [Deep Classifier Mimicry without Data Access](https://mlsanity.com/paper/2306.02090/) (2023)
- [Relation-Guided Adversarial Learning for Data-free Knowledge Transfer](https://mlsanity.com/paper/2412.11380/) (2024)
- [Tractable Representation Learning with Probabilistic Circuits](https://mlsanity.com/paper/2507.04385/) (2025)
- [DataFreeShield: Defending Adversarial Attacks without Training Data](https://mlsanity.com/paper/2406.15635/) (2024)
- [Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2608.22003/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) has enabled remarkable progress in model compression and knowledge transfer. However, KD requires a large volume of original data or their representation statistics that are not usually available in practice. Data-free KD has recently been proposed to resolve this problem, wherein teacher and student models are fed by a synthetic sample generator trained from the teacher. Nonetheless, existing data-free KD methods rely on fine-tuning of weights to balance multiple losses, and ignore the diversity of generated samples, resulting in limited accuracy and robustness. To overcome this challenge, we propose robustness and diversity seeking data-free KD (RDSKD) in this paper. The generator loss function is crafted to produce samples with high authenticity, class diversity, and inter-sample diversity. Without real data, the objectives of seeking high sample authenticity and class diversity often conflict with each other, causing frequent loss fluctuations. We mitigate this by exponentially penalizing loss increments. With MNIST, CIFAR-10, and SVHN datasets, our experiments show that RDSKD achieves higher accuracy with more robustness over different hyperparameter settings, compared to other data-free KD methods such as DAFL, MSKD, ZSKD, and DeepInversion.