# Undistillable: Making A Nasty Teacher That CANNOT teach students

arXiv: https://arxiv.org/abs/2105.07381 · HTML: https://mlsanity.com/paper/2105.07381/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 16 May 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Embedding Watermarks into Deep Neural Networks](https://mlsanity.com/paper/1701.04082/) (2017)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [PRADA: Protecting against DNN Model Stealing Attacks](https://mlsanity.com/paper/1805.02628/) (2018)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Is feature selection secure against training data poisoning?](https://mlsanity.com/paper/1804.07933/) (2018)
- [[Extended version] Rethinking Deep Neural Network Ownership Verification: Embedding Passports to Defeat Ambiguity Attacks](https://mlsanity.com/paper/1909.07830/) (2019)
- [Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks](https://mlsanity.com/paper/1906.10908/) (2019)
- [Defending Against Model Stealing Attacks with Adaptive Misinformation](https://mlsanity.com/paper/1911.07100/) (2019)
- [Model Watermarking for Image Processing Networks](https://mlsanity.com/paper/2002.11088/) (2020)

## Built on by (most cited first, 30 of 36)

- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)
- [End-to-end Spoken Conversational Question Answering: Task, Dataset and Model](https://mlsanity.com/paper/2204.14272/) (2022)
- [Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation](https://mlsanity.com/paper/2302.14290/) (2023)
- [Antidistillation Sampling](https://mlsanity.com/paper/2504.13146/) (2025)
- [You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership](https://mlsanity.com/paper/2111.00162/) (2021)
- [Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2207.12106/) (2022)
- [DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation](https://mlsanity.com/paper/2505.19504/) (2025)
- [Efficient One Pass Self-distillation with Zipf's Label Smoothing](https://mlsanity.com/paper/2207.12980/) (2022)
- [Safe Distillation Box](https://mlsanity.com/paper/2112.03695/) (2021)
- [EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System](https://mlsanity.com/paper/2206.13441/) (2022)
- [Ungeneralizable Examples](https://mlsanity.com/paper/2404.14016/) (2024)
- [Model Stealing for Any Low-Rank Language Model](https://mlsanity.com/paper/2411.07536/) (2024)
- [Distilling the Undistillable: Learning from a Nasty Teacher](https://mlsanity.com/paper/2210.11728/) (2022)
- [Towards Undistillable Models by Minimizing Conditional Mutual Information](https://mlsanity.com/paper/2507.00012/) (2025)
- [RedAct: Redacting Agent Capability Traces for Procedural Skill Protection](https://mlsanity.com/paper/2606.10813/) (2026)
- [Adversarial Sparse Teacher: Defense Against Distillation-Based Model Stealing Attacks Using Adversarial Examples](https://mlsanity.com/paper/2403.05181/) (2024)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning](https://mlsanity.com/paper/2409.19912/) (2024)
- [The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model](https://mlsanity.com/paper/2501.16226/) (2025)
- [Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures](https://mlsanity.com/paper/2510.16968/) (2025)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [Lightweight Self-Knowledge Distillation with Multi-source Information Fusion](https://mlsanity.com/paper/2305.09183/) (2023)
- [Model-Guardian: Protecting against Data-Free Model Stealing Using Gradient Representations and Deceptive Predictions](https://mlsanity.com/paper/2503.18081/) (2025)
- [Distillation Traps and Guards: A Calibration Knob for LLM Distillability](https://mlsanity.com/paper/2604.18963/) (2026)
- [Lossless Anti-Distillation Sampling](https://mlsanity.com/paper/2605.18829/) (2026)
- [Safeguarding Text-to-Image Generative Models Against Unauthorized Knowledge Distillation](https://mlsanity.com/paper/2605.22060/) (2026)
- [Semantics-Preserved Distortion for Personal Privacy Protection in Information Management](https://mlsanity.com/paper/2201.00965/) (2022)
- [OVLA: Neural Network Ownership Verification using Latent Watermarks](https://mlsanity.com/paper/2306.13215/) (2023)
- [Linearizing Models for Efficient yet Robust Private Inference](https://mlsanity.com/paper/2402.05521/) (2024)
- [I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks](https://mlsanity.com/paper/2508.21654/) (2025)

## Abstract (arXiv)

Knowledge Distillation (KD) is a widely used technique to transfer knowledge from pre-trained teacher models to (usually more lightweight) student models. However, in certain situations, this technique is more of a curse than a blessing. For instance, KD poses a potential risk of exposing intellectual properties (IPs): even if a trained machine learning model is released in 'black boxes' (e.g., as executable software or APIs without open-sourcing code), it can still be replicated by KD through imitating input-output behaviors. To prevent this unwanted effect of KD, this paper introduces and investigates a concept called Nasty Teacher: a specially trained teacher network that yields nearly the same performance as a normal one, but would significantly degrade the performance of student models learned by imitating it. We propose a simple yet effective algorithm to build the nasty teacher, called self-undermining knowledge distillation. Specifically, we aim to maximize the difference between the output of the nasty teacher and a normal pre-trained network. Extensive experiments on several datasets demonstrate that our method is effective on both standard KD and data-free KD, providing the desirable KD-immunity to model owners for the first time. We hope our preliminary study can draw more awareness and interest in this new practical problem of both social and legal importance.