# Safe Distillation Box

arXiv: https://arxiv.org/abs/2112.03695 · HTML: https://mlsanity.com/paper/2112.03695/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://mlsanity.com/paper/1902.11237/) (2019)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Clean-Label Backdoor Attacks on Video Recognition Models](https://mlsanity.com/paper/2003.03030/) (2020)
- [Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation](https://mlsanity.com/paper/1808.10307/) (2018)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [Backdoor Attacks to Graph Neural Networks](https://mlsanity.com/paper/2006.11165/) (2020)
- [Adversarial Frontier Stitching for Remote Neural Network Watermarking](https://mlsanity.com/paper/1711.01894/) (2017)
- [Incremental Learning Through Deep Adaptation](https://mlsanity.com/paper/1705.04228/) (2017)
- [Digital Watermarking for Deep Neural Networks](https://mlsanity.com/paper/1802.02601/) (2018)
- [Amalgamating Knowledge towards Comprehensive Classification](https://mlsanity.com/paper/1811.02796/) (2018)
- [Undistillable: Making A Nasty Teacher That CANNOT teach students](https://mlsanity.com/paper/2105.07381/) (2021)
- [Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN](https://mlsanity.com/paper/1711.07607/) (2017)

## Built on by (most cited first, 9 of 9)

- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Learning with Recoverable Forgetting](https://mlsanity.com/paper/2207.08224/) (2022)
- [Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation](https://mlsanity.com/paper/2302.14290/) (2023)
- [TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models](https://mlsanity.com/paper/2301.01296/) (2023)
- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [Data and Model Poisoning Backdoor Attacks on Wireless Federated Learning, and the Defense Mechanisms: A Comprehensive Survey](https://mlsanity.com/paper/2312.08667/) (2023)
- [Ungeneralizable Examples](https://mlsanity.com/paper/2404.14016/) (2024)
- [Partial Network Cloning](https://mlsanity.com/paper/2303.10597/) (2023)
- [Towards Few-Call Model Stealing via Active Self-Paced Knowledge Distillation and Diffusion-Based Image Generation](https://mlsanity.com/paper/2310.00096/) (2023)

## Abstract (arXiv)

Knowledge distillation (KD) has recently emerged as a powerful strategy to transfer knowledge from a pre-trained teacher model to a lightweight student, and has demonstrated its unprecedented success over a wide spectrum of applications. In spite of the encouraging results, the KD process per se poses a potential threat to network ownership protection, since the knowledge contained in network can be effortlessly distilled and hence exposed to a malicious user. In this paper, we propose a novel framework, termed as Safe Distillation Box (SDB), that allows us to wrap a pre-trained model in a virtual box for intellectual property protection. Specifically, SDB preserves the inference capability of the wrapped model to all users, but precludes KD from unauthorized users. For authorized users, on the other hand, SDB carries out a knowledge augmentation scheme to strengthen the KD performances and the results of the student model. In other words, all users may employ a model in SDB for inference, but only authorized users get access to KD from the model. The proposed SDB imposes no constraints over the model architecture, and may readily serve as a plug-and-play solution to protect the ownership of a pre-trained network. Experiments across various datasets and architectures demonstrate that, with SDB, the performance of an unauthorized KD drops significantly while that of an authorized gets enhanced, demonstrating the effectiveness of SDB.