# ProARD: progressive adversarial robustness distillation: provide wide range of robust students

arXiv: https://arxiv.org/abs/2506.07666 · HTML: https://mlsanity.com/paper/2506.07666/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 59 of its 78 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Attacks Which Do Not Kill Training Make Adversarial Learning Stronger](https://mlsanity.com/paper/2002.11242/) (2020)
- [Adversarial Logit Pairing](https://mlsanity.com/paper/1803.06373/) (2018)
- [Geometry-aware Instance-reweighted Adversarial Training](https://mlsanity.com/paper/2010.01736/) (2020)
- [Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets](https://mlsanity.com/paper/2002.05990/) (2020)
- [Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems](https://mlsanity.com/paper/1907.10456/) (2019)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [Learnable Boundary Guided Adversarial Training](https://mlsanity.com/paper/2011.11164/) (2020)
- [When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks](https://mlsanity.com/paper/1911.10695/) (2019)
- [Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better](https://mlsanity.com/paper/2108.07969/) (2021)
- [Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks](https://mlsanity.com/paper/2110.03825/) (2021)

## Abstract (arXiv)

Adversarial Robustness Distillation (ARD) has emerged as an effective method to enhance the robustness of lightweight deep neural networks against adversarial attacks. Current ARD approaches have leveraged a large robust teacher network to train one robust lightweight student. However, due to the diverse range of edge devices and resource constraints, current approaches require training a new student network from scratch to meet specific constraints, leading to substantial computational costs and increased CO2 emissions. This paper proposes Progressive Adversarial Robustness Distillation (ProARD), enabling the efficient one-time training of a dynamic network that supports a diverse range of accurate and robust student networks without requiring retraining. We first make a dynamic deep neural network based on dynamic layers by encompassing variations in width, depth, and expansion in each design stage to support a wide range of architectures. Then, we consider the student network with the largest size as the dynamic teacher network. ProARD trains this dynamic network using a weight-sharing mechanism to jointly optimize the dynamic teacher network and its internal student networks. However, due to the high computational cost of calculating exact gradients for all the students within the dynamic network, a sampling mechanism is required to select a subset of students. We show that random student sampling in each iteration fails to produce accurate and robust students.