mlsanityData to 14 Oct 2026

ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation

arXiv 2211.00239 ↗First posted 1 Nov 2022cs.LG cs.AIcited by 1

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Adversarial training is the most promising method for learning robust models against adversarial examples. A recent study has shown that knowledge distillation between the same architectures is effective in improving the performance of adversarial training. Exploiting knowledge distillation is a new approach to improve adversarial training and has attracted much attention. However, its performance is still insufficient. Therefore, we propose Adversarial Robust Distillation with Internal Representation~(ARDIR) to utilize knowledge distillation even more effectively. In addition to the output of the teacher model, ARDIR uses the internal representation of the teacher model as a label for adversarial training. This enables the student model to be trained with richer, more informative labels. As a result, ARDIR can learn more robust student models. We show that ARDIR outperforms previous methods in our experiments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 17 of its 23 references that are in the map. First posted on arXiv 1 Nov 2022.

Builds on

17 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better18 Aug
  2. 2021Reliable Adversarial Distillation with Unreliable Teachers9 Jun

Other references · 15

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  7. 2016Adversarial examples in the physical worldcited by 1,769
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  11. 2020Overfitting in adversarially robust deep learningcited by 506
  12. 2018Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applicationscited by 224
3 more
  1. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  2. 2019Adversarially Robust Distillationcited by 125
  3. 2017DeepSign: Deep Learning for Automatic Malware Signature Generation and Classificationcited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ProARD: progressive adversarial robustness distillation: provide wide range of robust studentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022On the benefits of knowledge distillation for adversarial robustness
  2. 2025Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
  3. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?
  4. 2022Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training
  5. 2026Learning Better Certified Models from Empirically-Robust Teachers
  6. 2026Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
  7. 2024Distilling Adversarial Robustness Using Heterogeneous Teachers
  8. 2022AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation