mlsanityData to 14 Oct 2026

Teach Me to Trick: Exploring Adversarial Transferability via Knowledge Distillation

arXiv 2507.21992 ↗First posted 29 Jul 2025cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

We investigate whether knowledge distillation (KD) from multiple heterogeneous teacher models can enhance the generation of transferable adversarial examples. A lightweight student model is trained using two KD strategies: curriculum-based switching and joint optimization, with ResNet50 and DenseNet-161 as teachers. The trained student is then used to generate adversarial examples using FG, FGS, and PGD attacks, which are evaluated against a black-box target model (GoogLeNet). Our results show that student models distilled from multiple teachers achieve attack success rates comparable to ensemble-based baselines, while reducing adversarial example generation time by up to a factor of six. An ablation study further reveals that lower temperature settings and the inclusion of hard-label supervision significantly enhance transferability. These findings suggest that KD can serve not only as a model compression technique but also as a powerful tool for improving the efficiency and effectiveness of black-box adversarial attacks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 11 of its 14 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

11 of 14 references.

References · 11

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2020Knowledge Distillation: A Surveycited by 1,603
  8. 2017Deep Mutual Learningcited by 628
  9. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  10. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  11. 2023Generalizable Black-Box Adversarial Attack with Meta Learningcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examples
  2. 2022On the benefits of knowledge distillation for adversarial robustness
  3. 2023Common Knowledge Learning for Generating Transferable Adversarial Examples
  4. 2026Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
  5. 2021Improved Knowledge Distillation via Adversarial Collaboration
  6. 2022ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation
  7. 2024Distilling Adversarial Robustness Using Heterogeneous Teachers
  8. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?