# Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

arXiv: https://arxiv.org/abs/1805.05532 · HTML: https://mlsanity.com/paper/1805.05532/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2018.

## Built on by (most cited first, 30 of 49)

- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [3D-to-2D Distillation for Indoor Scene Parsing](https://mlsanity.com/paper/2104.02243/) (2021)
- [Co-advise: Cross Inductive Bias Distillation](https://mlsanity.com/paper/2106.12378/) (2021)
- [Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples](https://mlsanity.com/paper/2111.02625/) (2021)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [Lion: Adversarial Distillation of Proprietary Large Language Models](https://mlsanity.com/paper/2305.12870/) (2023)
- [Delving into the Adversarial Robustness of Federated Learning](https://mlsanity.com/paper/2302.09479/) (2023)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [Self-Knowledge Distillation via Dropout](https://mlsanity.com/paper/2208.05642/) (2022)
- [Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis](https://mlsanity.com/paper/2104.04868/) (2021)
- [Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition](https://mlsanity.com/paper/2109.14710/) (2021)
- [NeRN -- Learning Neural Representations for Neural Networks](https://mlsanity.com/paper/2212.13554/) (2022)
- [Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack](https://mlsanity.com/paper/2104.05921/) (2021)
- [What Makes a Good Dataset for Knowledge Distillation?](https://mlsanity.com/paper/2411.12817/) (2024)
- [Cross-Class Feature Augmentation for Class Incremental Learning](https://mlsanity.com/paper/2304.01899/) (2023)
- [Rethinking Impersonation and Dodging Attacks on Face Recognition Systems](https://mlsanity.com/paper/2401.08903/) (2024)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation](https://mlsanity.com/paper/2303.11611/) (2023)
- [One-Shot Federated Learning for LEO Constellations that Reduces Convergence Time from Days to 90 Minutes](https://mlsanity.com/paper/2305.12316/) (2023)
- [Adversarial Examples for Good: Adversarial Examples Guided Imbalanced Learning](https://mlsanity.com/paper/2201.12356/) (2022)
- [Illumination Distillation Framework for Nighttime Person Re-Identification and A New Benchmark](https://mlsanity.com/paper/2308.16486/) (2023)
- [Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data](https://mlsanity.com/paper/2201.00111/) (2022)
- [Knowledge-guided Causal Intervention for Weakly-supervised Object Localization](https://mlsanity.com/paper/2301.01060/) (2023)
- [GRAN: Ghost Residual Attention Network for Single Image Super Resolution](https://mlsanity.com/paper/2302.14557/) (2023)
- [Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs](https://mlsanity.com/paper/2506.17231/) (2025)
- [Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systems](https://mlsanity.com/paper/2205.02604/) (2022)
- [ACQ: Improving Generative Data-free Quantization Via Attention Correction](https://mlsanity.com/paper/2301.07266/) (2023)
- [Evasion Adversarial Attacks Remain Impractical Against ML-based Network Intrusion Detection Systems, Especially Dynamic Ones](https://mlsanity.com/paper/2306.05494/) (2023)
- [Finding Patterns in Ambiguity: Interpretable Stress Testing in the Decision~Boundary](https://mlsanity.com/paper/2408.06302/) (2024)
- [Confidence Conditioned Knowledge Distillation](https://mlsanity.com/paper/2107.06993/) (2021)

## Abstract (arXiv)

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.