mlsanityData to 14 Oct 2026

ProARD: progressive adversarial robustness distillation: provide wide range of robust students

arXiv 2506.07666 ↗First posted 9 Jun 2025cs.LGcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Adversarial Robustness Distillation (ARD) has emerged as an effective method to enhance the robustness of lightweight deep neural networks against adversarial attacks. Current ARD approaches have leveraged a large robust teacher network to train one robust lightweight student. However, due to the diverse range of edge devices and resource constraints, current approaches require training a new student network from scratch to meet specific constraints, leading to substantial computational costs and increased CO2 emissions. This paper proposes Progressive Adversarial Robustness Distillation (ProARD), enabling the efficient one-time training of a dynamic network that supports a diverse range of accurate and robust student networks without requiring retraining. We first make a dynamic deep neural network based on dynamic layers by encompassing variations in width, depth, and expansion in each design stage to support a wide range of architectures. Then, we consider the student network with the largest size as the dynamic teacher network. ProARD trains this dynamic network using a weight-sharing mechanism to jointly optimize the dynamic teacher network and its internal student networks. However, due to the high computational cost of calculating exact gradients for all the students within the dynamic network, a sampling mechanism is required to select a subset of students. We show that random student sampling in each iteration fails to produce accurate and robust students.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 59 of its 78 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

59 of 78 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor11 Mar
  2. 2024Distilling Adversarial Robustness Using Heterogeneous Teachers23 Feb
  3. 2023NEO-KD: Knowledge-Distillation-Based Adversarial Training for Robust Multi-Exit Neural Networks1 Nov
  4. 2022ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation1 Nov
  5. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?22 Oct
  6. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better18 Aug
  7. 2021Reliable Adversarial Distillation with Unreliable Teachers9 Jun

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  10. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  11. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  12. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
38 more
  1. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  2. 2019Adversarial Training for Free!cited by 643
  3. 2023Segment Everything Everywhere All at Oncecited by 548
  4. 2019Object Detection in 20 Years: A Surveycited by 531
  5. 2020Overfitting in adversarially robust deep learningcited by 506
  6. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  7. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 243
  8. 2018Adversarial Logit Pairingcited by 224
  9. 2020Geometry-aware Instance-reweighted Adversarial Trainingcited by 198
  10. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 167
  11. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 153
  12. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  13. 2019Adversarially Robust Distillationcited by 125
  14. 2021On the Adversarial Robustness of Vision Transformerscited by 106
  15. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  16. 2019When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attackscited by 87
  17. 2021Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networkscited by 74
  18. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  19. 2021Anycost GANs for Interactive Image Synthesis and Editingcited by 54
  20. 2022Understanding Robust Overfitting of Adversarial Training and Beyondcited by 51
  21. 2021AdvRush: Searching for Adversarially Robust Neural Architecturescited by 28
  22. 2022Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspectivecited by 28
  23. 2020DSRNA: Differentiable Search of Robust Neural Architecturescited by 27
  24. 2021CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deploymentcited by 26
  25. 2021Network Augmentation for Tiny Deep Learningcited by 24
  26. 2020Adversarially Robust Neural Architecturescited by 22
  27. 2020Anti-Bandit Neural Architecture Search for Model Defensecited by 16
  28. 2022Lite Pose: Efficient Architecture Design for 2D Human Pose Estimationcited by 16
  29. 2023Neural Architecture Design and Robustness: A Datasetcited by 15
  30. 2021Neural Architecture Dilation for Adversarial Robustnesscited by 13
  31. 2021Learning Diverse-Structured Networks for Adversarial Robustnesscited by 12
  32. 2020Effective, Efficient and Robust Neural Architecture Searchcited by 11
  33. 2023Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbationscited by 6
  34. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  35. 2023DetOFA: Efficient Training of Once-for-All Networks for Object Detection Using Path Filtercited by 5
  36. 2023Robust Neural Architecture Searchcited by 5
  37. 2021Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Searchcited by 3
  38. 2024Towards Accurate and Robust Architectures via Neural Architecture Searchcited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022On the benefits of knowledge distillation for adversarial robustness
  2. 2025Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
  3. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation
  4. 2026CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
  5. 2026Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
  6. 2023Improving Defensive Distillation using Teacher Assistant
  7. 2026Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
  8. 2026Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers