mlsanityData to 14 Oct 2026

PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor

arXiv 2403.06668 ↗First posted 11 Mar 2024cs.LG cs.CVcited by 7

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Adversarial robustness of the neural network is a significant concern when it is applied to security-critical domains. In this situation, adversarial distillation is a promising option which aims to distill the robustness of the teacher network to improve the robustness of a small student network. Previous works pretrain the teacher network to make it robust against the adversarial examples aimed at itself. However, the adversarial examples are dependent on the parameters of the target network. The fixed teacher network inevitably degrades its robustness against the unseen transferred adversarial examples which target the parameters of the student network in the adversarial distillation process. We propose PeerAiD to make a peer network learn the adversarial examples of the student network instead of adversarial examples aimed at itself. PeerAiD is an adversarial distillation that trains the peer network and the student network simultaneously in order to specialize the peer network for defending the student network. We observe that such peer networks surpass the robustness of the pretrained robust teacher model against adversarial examples aimed at the student network. With this peer network and adversarial distillation, PeerAiD achieves significantly higher robustness of the student network with AutoAttack (AA) accuracy by up to 1.66%p and improves the natural accuracy of the student network by up to 4.72%p with ResNet-18 on TinyImageNet dataset. Code is available at https://github.com/jaewonalive/PeerAiD.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

38 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better18 Aug
  2. 2021Reliable Adversarial Distillation with Unreliable Teachers9 Jun

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
24 more
  1. 2016Wide Residual Networkscited by 3,070
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  8. 2018Robustness May Be at Odds with Accuracycited by 880
  9. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  10. 2019Adversarial Training for Free!cited by 643
  11. 2017Deep Mutual Learningcited by 628
  12. 2020Overfitting in adversarially robust deep learningcited by 506
  13. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  14. 2020Understanding and Improving Fast Adversarial Trainingcited by 214
  15. 2018Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Carscited by 200
  16. 2019Adversarially Robust Distillationcited by 125
  17. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  18. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  19. 2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
  20. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  21. 2021Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfittingcited by 14
  22. 2023Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averagingcited by 14
  23. 2020Adversarial Training with Stochastic Weight Averagecited by 12
  24. 2021Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compressioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  2. 2025LISArD: Learning Image Similarity to Defend Against Gray-box Adversarial Attackscited by 3
  3. 2025Unsupervised Robust Domain Adaptation: Paradigm, Theory and Algorithmcited by 3
  4. 2025SC-Pro: Training-Free Framework for Defending Unsafe Image Synthesis Attackcited by 1
  5. 2025ProARD: progressive adversarial robustness distillation: provide wide range of robust studentscited by 0
  6. 2025Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillationcited by 0
  7. 2026CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Mutual Adversarial Training: Learning together is better than going alone
  2. 2022On the benefits of knowledge distillation for adversarial robustness
  3. 2026Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
  4. 2023Improving Defensive Distillation using Teacher Assistant
  5. 2024Distilling Adversarial Robustness Using Heterogeneous Teachers
  6. 2026Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
  7. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation
  8. 2025Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation