# Knowledge Distillation Beyond Model Compression

arXiv: https://arxiv.org/abs/2007.01922 · HTML: https://mlsanity.com/paper/2007.01922/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Distill on the Go: Online knowledge distillation in self-supervised learning](https://mlsanity.com/paper/2104.09866/) (2021)
- [Understanding Self-Distillation in the Presence of Label Noise](https://mlsanity.com/paper/2301.13304/) (2023)
- [PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation](https://mlsanity.com/paper/2208.05669/) (2022)
- [Sparse Coding in a Dual Memory System for Lifelong Learning](https://mlsanity.com/paper/2301.05058/) (2022)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [Fast Quantum Convolutional Neural Networks for Low-Complexity Object Detection in Autonomous Driving Applications](https://mlsanity.com/paper/2401.01370/) (2023)
- [LILA-BOTI : Leveraging Isolated Letter Accumulations By Ordering Teacher Insights for Bangla Handwriting Recognition](https://mlsanity.com/paper/2205.11420/) (2022)
- [Rotation Invariant Quantization for Model Compression](https://mlsanity.com/paper/2303.03106/) (2023)
- [NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning](https://mlsanity.com/paper/2402.14139/) (2024)
- [Mixed Sample Augmentation for Online Distillation](https://mlsanity.com/paper/2206.12370/) (2022)
- [Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging](https://mlsanity.com/paper/2212.02931/) (2022)
- [A Layered Self-Supervised Knowledge Distillation Framework for Efficient Multimodal Learning on the Edge](https://mlsanity.com/paper/2506.07055/) (2025)
- [A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs](https://mlsanity.com/paper/2608.08294/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) is commonly deemed as an effective model compression technique in which a compact model (student) is trained under the supervision of a larger pretrained model or an ensemble of models (teacher). Various techniques have been proposed since the original formulation, which mimic different aspects of the teacher such as the representation space, decision boundary, or intra-data relationship. Some methods replace the one-way knowledge distillation from a static teacher with collaborative learning between a cohort of students. Despite the recent advances, a clear understanding of where knowledge resides in a deep neural network and an optimal method for capturing knowledge from teacher and transferring it to student remains an open question. In this study, we provide an extensive study on nine different KD methods which covers a broad spectrum of approaches to capture and transfer knowledge. We demonstrate the versatility of the KD framework on different datasets and network architectures under varying capacity gaps between the teacher and student. The study provides intuition for the effects of mimicking different aspects of the teacher and derives insights from the performance of the different distillation approaches to guide the design of more effective KD methods. Furthermore, our study shows the effectiveness of the KD framework in learning efficiently under varying severity levels of label noise and class imbalance, consistently providing generalization gains over standard training. We emphasize that the efficacy of KD goes much beyond a model compression technique and it should be considered as a general-purpose training paradigm which offers more robustness to common challenges in the real-world datasets compared to the standard training procedure.