Data to 14 Oct 2026

Reliable Adversarial Distillation with Unreliable Teachers

arXiv 2106.04928 ↗First posted 9 Jun 2021cs.LGcited by 56

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

In ordinary distillation, student networks are trained with soft labels (SLs) given by pretrained teacher networks, and students are expected to improve upon teachers since SLs are stronger supervision than the original hard labels. However, when considering adversarial robustness, teachers may become unreliable and adversarial distillation may not work: teachers are pretrained on their own adversarial data, and it is too demanding to require that teachers are also good at every adversarial data queried by students. Therefore, in this paper, we propose reliable introspective adversarial distillation (IAD) where students partially instead of fully trust their teachers. Specifically, IAD distinguishes between three cases given a query of a natural data (ND) and the corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL is fully trusted; (b) if a teacher is good at ND but not AD, its SL is partially trusted and the student also takes its own SL into account; (c) otherwise, the student only relies on its own SL. Experiments demonstrate the effectiveness of IAD for improving upon teachers in terms of adversarial robustness.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 30 of its 41 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

30 of 41 references.

References · 30

2015Deep Residual Learning for Image Recognitioncited by 52,096
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2015Distilling the Knowledge in a Neural Networkcited by 10,324
2014Explaining and Harnessing Adversarial Examplescited by 7,364
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
2013Intriguing properties of neural networkscited by 5,155
2016Wide Residual Networkscited by 3,070
2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
2020Overfitting in adversarially robust deep learningcited by 506

18 more

2019Unlabeled Data Improves Adversarial Robustnesscited by 416
2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 243
2020Geometry-aware Instance-reweighted Adversarial Trainingcited by 198
2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 172
2019Are Labels Required for Improving Adversarial Robustness?cited by 163
2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
2019Adversarially Robust Distillationcited by 125
2020Adversarial Machine Learning -- Industry Perspectivescited by 122
2018Curriculum Adversarial Trainingcited by 89
2021Improving Adversarial Robustness via Channel-wise Activation Suppressingcited by 76
2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
2020Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Freecited by 48
2021Device-Cloud Collaborative Learning for Recommendationcited by 34
2021Analysis and Applications of Class-wise Robustness in Adversarial Trainingcited by 33
2020Encoding Robustness to Image Style via Adversarial Feature Perturbationscited by 18
2021Learning Diverse-Structured Networks for Adversarial Robustnesscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 17; 2024: 11; 2025: 15; 2026: 5.

56 papers.

Sort:

By area:

2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 79
2022On the benefits of knowledge distillation for adversarial robustnesscited by 18
2023Survey on Adversarial Attack and Defense for Medical Image Analysis: Methods and Challengescited by 17
2021LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Trainingcited by 13
2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
2022Visual Adversarial Attacks and Defenses in the Physical World: A Surveycited by 11
2023Adversarial Contrastive Distillation with Adaptive Denoisingcited by 9
2023AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Frameworkcited by 9
2023Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillationcited by 8
2022Accelerating Certified Robustness Training via Knowledge Transfercited by 7
2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
2023Annealing Self-Distillation Rectification Improves Adversarial Trainingcited by 7

38 more

2023Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoffcited by 7
2024Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-offcited by 7
2024PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutorcited by 7
2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 6
2023Data-Free Hard-Label Robustness Stealing Attackcited by 6
2024Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Trainingcited by 6
2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
2024Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attackscited by 5
2024Improve Knowledge Distillation via Label Revision and Data Selectioncited by 5
2024Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledgecited by 5
2025Long-tailed Adversarial Training with Self-Distillationcited by 5
2023Combating Exacerbated Heterogeneity for Robust Models in Federated Learningcited by 4
2023Exploring Robust Features for Improving Adversarial Robustnesscited by 4
2023NEO-KD: Knowledge-Distillation-Based Adversarial Training for Robust Multi-Exit Neural Networkscited by 3
2024Distilling Adversarial Robustness Using Heterogeneous Teacherscited by 3
2024Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replaycited by 3
2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
2025LISArD: Learning Image Similarity to Defend Against Gray-box Adversarial Attackscited by 3
2025Unsupervised Robust Domain Adaptation: Paradigm, Theory and Algorithmcited by 3
2023CAT:Collaborative Adversarial Trainingcited by 2
2024Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teacherscited by 2
2025Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devicescited by 2
2025Continual Low-Rank Adapters for LLM-based Generative Recommender Systemscited by 2
2022DAFT: Distilling Adversarially Fine-tuned Models for Better OOD Generalizationcited by 1
2022ARDIR: Improving Robustness using Knowledge Distillation of Internal Representationcited by 1
2023Faithful Knowledge Distillationcited by 1
2025CIARD: Cyclic Iterative Adversarial Robustness Distillationcited by 1
2026Learning Better Certified Models from Empirically-Robust Teacherscited by 1
2022Towards Robust Stacked Capsule Autoencoder with Hybrid Adversarial Trainingcited by 0
2023AdvFunMatch: When Consistent Teaching Meets Adversarial Robustnesscited by 0
2023DAD++: Improved Data-free Test Time Adversarial Defensecited by 0
2025MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillationcited by 0
2025Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Bettercited by 0
2025Identifying and Understanding Cross-Class Features in Adversarial Trainingcited by 0
2025ProARD: progressive adversarial robustness distillation: provide wide range of robust studentscited by 0
2025SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defensecited by 0
2025Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillationcited by 0
2025Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillationcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way