mlsanityData to 14 Oct 2026

How to Backdoor Image Knowledge Distillation

arXiv 2504.21323 ↗First posted 30 Apr 2025cs.CR cs.AI cs.LGcited by 1

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Knowledge distillation is widely used to transfer behavior from a large teacher model to a smaller student. It is often assumed to be safe when the teacher is clean, because classic backdoor attacks rely on poisoned labels and triggers in supervised training, whereas distillation trains the student to match a teacher's outputs. We show that this assumption can fail when the distillation dataset itself is poisoned. Our attack injects triggered and manipulated images that a clean teacher already predicts as an attacker chosen target label, which causes the student to learn a backdoor even though the teacher remains unaffected. We evaluate this threat across multiple manipulation strategies, including targeted adversarial perturbations and targeted GAN based class transitions, and study how distillation settings influence both accuracy and attack success. We show that the attack remains effective even at a 10% poisoning rates. The results demonstrate that a clean teacher alone is not a sufficient safeguard: poisoned distillation data can produce a strongly backdoored student while maintaining competitive performance on clean images. These findings show that the integrity and provenance of distillation data are part of the security boundary of data intensive KD pipelines, even when the teacher itself is trusted.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 29 of its 39 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

29 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teachers4 Jun

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  9. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  10. 2017Deep Mutual Learningcited by 628
  11. 2017One pixel attack for fooling deep neural networkscited by 582
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
16 more
  1. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  2. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
  3. 2018Generating Adversarial Examples with Adversarial Networkscited by 258
  4. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
  5. 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
  6. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  7. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  8. 2018Adversarial examples from computational constraintscited by 80
  9. 2017Adversarial Transformation Networks: Learning to Generate Adversarial Examplescited by 62
  10. 2018Private Model Compression via Knowledge Distillationcited by 37
  11. 2021Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attackscited by 33
  12. 2022Pixle: a fast and effective black-box attack based on rearranging pixelscited by 20
  13. 2020Patch-wise++ Perturbation for Adversarial Targeted Attackscited by 19
  14. 2019Comment on "Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network"cited by 12
  15. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 9
  16. 2023Universal Backdoor Attackscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Distillation Mitigate Backdoor Attacks in Pre-trained Encoders?
  2. 2025Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack
  3. 2023Backdoor Attacks Against Dataset Distillation
  4. 2023INK: Inheritable Natural Backdoor Attack Against Model Distillation
  5. 2025Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation
  6. 2022On the benefits of knowledge distillation for adversarial robustness
  7. 2021Excess Capacity and Backdoor Poisoning
  8. 2025Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation