# Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation

arXiv: https://arxiv.org/abs/2303.11611 · HTML: https://mlsanity.com/paper/2303.11611/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [ComDefend: An Efficient Image Compression Model to Defend Adversarial Examples](https://mlsanity.com/paper/1811.12673/) (2018)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better](https://mlsanity.com/paper/2108.07969/) (2021)
- [Curriculum Learning of Multiple Tasks](https://mlsanity.com/paper/1412.1353/) (2014)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Reliable Adversarial Distillation with Unreliable Teachers](https://mlsanity.com/paper/2106.04928/) (2021)
- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [Knowledge Distillation with Adversarial Samples Supporting Decision Boundary](https://mlsanity.com/paper/1805.05532/) (2018)

## Built on by (most cited first, 7 of 7)

- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [Sampling to Distill: Knowledge Transfer from Open-World Data](https://mlsanity.com/paper/2307.16601/) (2023)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Improve Knowledge Distillation via Label Revision and Data Selection](https://mlsanity.com/paper/2404.03693/) (2024)
- [HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images](https://mlsanity.com/paper/2403.07912/) (2024)
- [Self-Cooperation Knowledge Distillation for Novel Class Discovery](https://mlsanity.com/paper/2407.01930/) (2024)
- [FERD: Fairness-Enhanced Data-Free Robustness Distillation](https://mlsanity.com/paper/2509.20793/) (2025)

## Abstract (arXiv)

Adversarial Robustness Distillation (ARD) is a promising task to solve the issue of limited adversarial robustness of small capacity models while optimizing the expensive computational costs of Adversarial Training (AT). Despite the good robust performance, the existing ARD methods are still impractical to deploy in natural high-security scenes due to these methods rely entirely on original or publicly available data with a similar distribution. In fact, these data are almost always private, specific, and distinctive for scenes that require high robustness. To tackle these issues, we propose a challenging but significant task called Data-Free Adversarial Robustness Distillation (DFARD), which aims to train small, easily deployable, robust models without relying on data. We demonstrate that the challenge lies in the lower upper bound of knowledge transfer information, making it crucial to mining and transferring knowledge more efficiently. Inspired by human education, we design a plug-and-play Interactive Temperature Adjustment (ITA) strategy to improve the efficiency of knowledge transfer and propose an Adaptive Generator Balance (AGB) module to retain more data information. Our method uses adaptive hyperparameters to avoid a large number of parameter tuning, which significantly outperforms the combination of existing techniques. Meanwhile, our method achieves stable and reliable performance on multiple benchmarks.