# Revisiting Data-Free Knowledge Distillation with Poisoned Teachers arXiv: https://arxiv.org/abs/2306.02368 ยท HTML: https://mlsanity.com/paper/2306.02368/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Machine unlearning, one of 12 directions in LLM safety and security. - Builds on 44 of its 63 references that are in the map. - First posted on arXiv 4 Jun 2023. ## Placed in - LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/) - Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/) ## Builds on (most cited first, 30 of 44) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016) - [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016) - [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021) - [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022) - [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017) - [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018) - [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021) - [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019) - [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018) - [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022) - [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021) - [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020) - [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021) - [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020) - [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020) - [Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks](https://mlsanity.com/paper/2101.05930/) (2021) - [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019) - [Anti-Backdoor Learning: Training Clean Models on Poisoned Data](https://mlsanity.com/paper/2110.11571/) (2021) - [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021) - [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://mlsanity.com/paper/1902.11237/) (2019) - [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019) - [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020) - [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021) - [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022) - [Adversarial Unlearning of Backdoors via Implicit Hypergradient](https://mlsanity.com/paper/2110.03735/) (2021) - [Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective](https://mlsanity.com/paper/2104.03413/) (2021) - [On the Iteration Complexity of Hypergradient Computation](https://mlsanity.com/paper/2006.16218/) (2020) - [Entangled Watermarks as a Defense against Model Extraction](https://mlsanity.com/paper/2002.12200/) (2020) ## Built on by (most cited first, 9 of 9) - [Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation](https://mlsanity.com/paper/2410.14425/) (2024) - [Cascading Adversarial Bias from Injection to Distillation in Language Models](https://mlsanity.com/paper/2505.24842/) (2025) - [Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation](https://mlsanity.com/paper/2402.12406/) (2024) - [When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need](https://mlsanity.com/paper/2507.04119/) (2025) - [Safe and Robust Watermark Injection with a Single OoD Image](https://mlsanity.com/paper/2309.01786/) (2023) - [HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning](https://mlsanity.com/paper/2409.19912/) (2024) - [Evading Data Provenance in Deep Neural Networks](https://mlsanity.com/paper/2508.01074/) (2025) - [How to Backdoor Image Knowledge Distillation](https://mlsanity.com/paper/2504.21323/) (2025) - [Fusing Pruned and Backdoored Models: Optimal Transport-based Data-free Backdoor Mitigation](https://mlsanity.com/paper/2408.15861/) (2024) ## Abstract (arXiv) Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model (known as the teacher model) to a smaller model (known as the student model) without access to the original training data used for training the teacher model. However, the security of the synthetic or out-of-distribution (OOD) data required in data-free KD is largely unknown and under-explored. In this work, we make the first effort to uncover the security risk of data-free KD w.r.t. untrusted pre-trained models. We then propose Anti-Backdoor Data-Free KD (ABD), the first plug-in defensive method for data-free KD methods to mitigate the chance of potential backdoors being transferred. We empirically evaluate the effectiveness of our proposed ABD in diminishing transferred backdoor knowledge while maintaining compatible downstream performances as the vanilla KD. We envision this work as a milestone for alarming and mitigating the potential backdoors in data-free KD. Codes are released at https://github.com/illidanlab/ABD.