mlsanityData to 13 Oct 2026

Revisiting Data-Free Knowledge Distillation with Poisoned Teachers

arXiv 2306.02368 ↗First posted 4 Jun 2023cs.LG cs.CRcited by 9

On the map: LLM safety and security → Machine unlearning

also in 1 more
Abstract (arXiv)

Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model (known as the teacher model) to a smaller model (known as the student model) without access to the original training data used for training the teacher model. However, the security of the synthetic or out-of-distribution (OOD) data required in data-free KD is largely unknown and under-explored. In this work, we make the first effort to uncover the security risk of data-free KD w.r.t. untrusted pre-trained models. We then propose Anti-Backdoor Data-Free KD (ABD), the first plug-in defensive method for data-free KD methods to mitigate the chance of potential backdoors being transferred. We empirically evaluate the effectiveness of our proposed ABD in diminishing transferred backdoor knowledge while maintaining compatible downstream performances as the vanilla KD. We envision this work as a milestone for alarming and mitigating the potential backdoors in data-free KD. Codes are released at https://github.com/illidanlab/ABD.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 44 of its 63 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

44 of 63 references.

References · 44

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  3. 2016Wide Residual Networkscited by 3,063
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,470
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,442
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  7. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,047
  8. 2018How To Backdoor Federated Learningcited by 764
  9. 2021The Medical Segmentation Decathloncited by 622
  10. 2019Meta-Learning with Implicit Gradientscited by 523
  11. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  12. 2018Spectral Signatures in Backdoor Attackscited by 481
32 more
  1. 2021Towards Personalized Federated Learningcited by 463
  2. 2020Backdoor Learning: A Surveycited by 424
  3. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  4. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 352
  5. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  6. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 323
  7. 2019Label-Consistent Backdoor Attackscited by 294
  8. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 262
  9. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  10. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 241
  11. 2019Data-Free Learning of Student Networkscited by 203
  12. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  13. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  14. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  15. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 164
  16. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspectivecited by 161
  17. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  18. 2020Entangled Watermarks as a Defense against Model Extractioncited by 149
  19. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  20. 2021SPECTRE: Defending Against Backdoor Attacks Using Robust Statisticscited by 128
  21. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 123
  22. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  23. 2022Narcissus: A Practical Clean-Label Backdoor Attack with Limited Informationcited by 102
  24. 2021Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratchcited by 101
  25. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  26. 2019How to Prove Your Model Belongs to You: A Blind-Watermark based Framework to Protect Intellectual Property of DNNcited by 66
  27. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  28. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  29. 2020Federated Knowledge Distillationcited by 51
  30. 2022PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learningcited by 34
  31. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  32. 2022Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetworkcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 4.

9 papers.

  1. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 26
  2. 2025Cascading Adversarial Bias from Injection to Distillation in Language Modelscited by 9
  3. 2024Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillationcited by 5
  4. 2025When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Needcited by 5
  5. 2023Safe and Robust Watermark Injection with a Single OoD Imagecited by 3
  6. 2024HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learningcited by 2
  7. 2025Evading Data Provenance in Deep Neural Networkscited by 2
  8. 2025How to Backdoor Image Knowledge Distillationcited by 1
  9. 2024Fusing Pruned and Backdoored Models: Optimal Transport-based Data-free Backdoor Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation
  2. 2025BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuning
  3. 2025Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack
  4. 2025Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks
  5. 2023Students Parrot Their Teachers: Membership Inference on Model Distillation
  6. 2025On Membership Inference Attacks in Knowledge Distillation
  7. 2022A Knowledge Distillation-Based Backdoor Attack in Federated Learning
  8. 2025Efficient Verified Machine Unlearning For Distillation