CIARD: Cyclic Iterative Adversarial Robustness Distillation
Not on the map. Why?
Abstract (arXiv)
Adversarial robustness distillation (ARD) aims to transfer both performance and robustness from teacher model to lightweight student model, enabling resilient performance on resource-constrained scenarios. Though existing ARD approaches enhance student model's robustness, the inevitable by-product leads to the degraded performance on clean examples. We summarize the causes of this problem inherent in existing methods with dual-teacher framework as: 1. The divergent optimization objectives of dual-teacher models, i.e., the clean and robust teachers, impede effective knowledge transfer to the student model, and 2. The iteratively generated adversarial examples during training lead to performance deterioration of the robust teacher model. To address these challenges, we propose a novel Cyclic Iterative ARD (CIARD) method with two key innovations: a. A multi-teacher framework with contrastive push-loss alignment to resolve conflicts in dual-teacher optimization objectives, and b. Continuous adversarial retraining to maintain dynamic teacher robustness against performance degradation from the varying adversarial examples. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CIARD achieves remarkable performance with an average 3.53 improvement in adversarial defense rates across various attack scenarios and a 5.87 increase in clean sample accuracy, establishing a new benchmark for balancing model robustness and generalization. Our code is available at https://github.com/eminentgu/CIARD
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 16 Sep 2025.
Builds on
39 of 51 references.
References · 39
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2013Intriguing properties of neural networkscited by 5,155
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2016Wide Residual Networkscited by 3,070
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
- 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
27 more
- 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
- 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 282
- 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
- 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 153
- 2019Adversarially Robust Distillationcited by 125
- 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 95
- 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
- 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 79
- 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
- 2019Adversarially robust transfer learningcited by 67
- 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
- 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 56
- 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
- 2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 35
- 2019What it Thinks is Important is Important: Robustness Transfers through Input Gradientscited by 34
- 2023A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentationcited by 33
- 2021A Review of Adversarial Attack and Defense for Classification Methodscited by 29
- 2022Towards Compositional Adversarial Robustness: Generalizing Adversarial Training to Composite Semantic Perturbationscited by 26
- 2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
- 2017Residual Convolutional CTC Networks for Automatic Speech Recognitioncited by 21
- 2022On the benefits of knowledge distillation for adversarial robustnesscited by 18
- 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
- 2024Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillationcited by 10
- 2023Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworkscited by 8
- 2023Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillationcited by 8
- 2022Transferring Adversarial Robustness Through Robust Representation Matchingcited by 7
- 2024Enhancing Content-based Recommendation via Large Language Modelcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.