mlsanityData to 14 Oct 2026

CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation

arXiv 2610.19911 ↗First posted 14 Oct 2026cs.LG cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods also fits: On-policy and self-distillation for LLMs

Abstract (arXiv)

Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong feedback transfer and derive a distribution-level upper bound on the refined teacher's risk. Experiments on CIFAR-10, CIFAR-100, and TinyImageNet-10 across multiple teacher-student architectures and base distillation methods show that CARD improves adversarial robustness in most evaluated settings while generally preserving clean accuracy. These results show that the distilled student can provide useful feedback to the teacher, enabling robustness transfer to proceed beyond the conventional one-way teacher-to-student paradigm.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 26 of its 41 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

26 of 41 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2016Wide Residual Networkscited by 3,070
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
14 more
  1. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 243
  2. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  3. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  4. 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 95
  5. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 79
  6. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 56
  7. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  8. 2022The Enemy of My Enemy is My Friend: Exploring Inverse Adversaries for Improving Adversarial Trainingcited by 23
  9. 2023Focus on Hiders: Exploring Hidden Threats for Enhancing Adversarial Trainingcited by 9
  10. 2023Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoffcited by 7
  11. 2024Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-offcited by 7
  12. 2024PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutorcited by 7
  13. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 6
  14. 2025CIARD: Cyclic Iterative Adversarial Robustness Distillationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
  2. 2026Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
  3. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation
  4. 2022On the benefits of knowledge distillation for adversarial robustness
  5. 2025Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
  6. 2023Faithful Knowledge Distillation
  7. 2023AdvFunMatch: When Consistent Teaching Meets Adversarial Robustness
  8. 2021Understanding Robustness in Teacher-Student Setting: A New Perspective