mlsanityData to 7 Oct 2026

Adversarially Robust Distillation

arXiv 1905.09747 ↗First posted 23 May 2019cs.LG cs.CV stat.MLcited by 124

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture in terms of robust accuracy, surpassing state-of-the-art methods on standard robustness benchmarks. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 26; 2023: 24; 2024: 26; 2025: 26; 2026: 5.

124 papers.

Sort:

By area:

  1. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  2. 2021Does Knowledge Distillation Really Work?cited by 187
  3. 2021Adversarial Examples Make Strong Poisonscited by 115
  4. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
  5. 2021Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networkscited by 73
  6. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  7. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
  8. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  9. 2022Fooling the Eyes of Autonomous Vehicles: Robust Physical Adversarial Examples Against Traffic Sign Recognition Systemscited by 27
  10. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 26
  11. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  12. 2021Adversarial Robustness for Unsupervised Domain Adaptationcited by 24
38 more
  1. 2025Antidistillation Samplingcited by 24
  2. 2022Can Model Compression Improve NLP Fairnesscited by 18
  3. 2022Eliminating Backdoor Triggers for Deep Neural Networks Using Attention Relation Graph Distillationcited by 18
  4. 2021Mutual Adversarial Training: Learning together is better than going alonecited by 17
  5. 2022On the benefits of knowledge distillation for adversarial robustnesscited by 17
  6. 2023Survey on Adversarial Attack and Defense for Medical Image Analysis: Methods and Challengescited by 16
  7. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?cited by 15
  8. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
  9. 2022Pruning Adversarially Robust Neural Networks without Adversarial Examplescited by 12
  10. 2022Visual Adversarial Attacks and Defenses in the Physical World: A Surveycited by 11
  11. 2023Exact Count of Boundary Pieces of ReLU Classifiers: Towards the Proper Complexity Measure for Classificationcited by 11
  12. 2021MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Mapscited by 9
  13. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillationcited by 9
  14. 2023Adversarial Contrastive Distillation with Adaptive Denoisingcited by 9
  15. 2023Robust Spatiotemporal Traffic Forecasting with Reinforced Dynamic Adversarial Trainingcited by 9
  16. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  17. 2024Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinementcited by 9
  18. 2022A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunitiescited by 8
  19. 2023Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillationcited by 8
  20. 2024DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillationcited by 8
  21. 2021Domain Invariant Adversarial Learningcited by 7
  22. 2022Transferring Adversarial Robustness Through Robust Representation Matchingcited by 7
  23. 2022Accelerating Certified Robustness Training via Knowledge Transfercited by 7
  24. 2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
  25. 2023Annealing Self-Distillation Rectification Improves Adversarial Trainingcited by 7
  26. 2023Data-Free Hard-Label Robustness Stealing Attackcited by 6
  27. 2024Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Trainingcited by 6
  28. 2024PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutorcited by 6
  29. 2024Adversarial Training: A Surveycited by 6
  30. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  31. 2022Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Datacited by 5
  32. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 5
  33. 2024Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attackscited by 5
  34. 2024Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Studycited by 5
  35. 2024Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledgecited by 5
  36. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  37. 2025Long-tailed Adversarial Training with Self-Distillationcited by 5
  38. 2021Unsupervised Domain Expansion for Visual Categorizationcited by 4

50 most cited of 124 All 124 (JSON)