# Modeling Teacher-Student Techniques in Deep Neural Networks for Knowledge Distillation

arXiv: https://arxiv.org/abs/1912.13179 · HTML: https://mlsanity.com/paper/1912.13179/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2019.

## Built on by (most cited first, 11 of 11)

- [MultiTask-CenterNet (MCN): Efficient and Diverse Multitask Learning using an Anchor Free Approach](https://mlsanity.com/paper/2108.05060/) (2021)
- [NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networks](https://mlsanity.com/paper/2401.13330/) (2024)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Long Tail Image Generation Through Feature Space Augmentation and Iterated Learning](https://mlsanity.com/paper/2405.01705/) (2024)
- [Using predefined vector systems as latent space configuration for neural network supervised training on data with arbitrarily large number of classes](https://mlsanity.com/paper/2510.04090/) (2025)
- [Marginally calibrated response distributions for end-to-end learning in autonomous driving](https://mlsanity.com/paper/2110.01050/) (2021)
- [Performance of computer vision algorithms for fine-grained classification using crowdsourced insect images](https://mlsanity.com/paper/2404.03474/) (2024)
- [2-speed network ensemble for efficient classification of incremental land-use/land-cover satellite image chips](https://mlsanity.com/paper/2203.08267/) (2022)
- [Self-supervised cross-modality learning for uncertainty-aware object detection and recognition in applications which lack pre-labelled training data](https://mlsanity.com/paper/2411.03082/) (2024)
- [HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Search](https://mlsanity.com/paper/2605.04103/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) is a new method for transferring knowledge of a structure under training to another one. The typical application of KD is in the form of learning a small model (named as a student) by soft labels produced by a complex model (named as a teacher). Due to the novel idea introduced in KD, recently, its notion is used in different methods such as compression and processes that are going to enhance the model accuracy. Although different techniques are proposed in the area of KD, there is a lack of a model to generalize KD techniques. In this paper, various studies in the scope of KD are investigated and analyzed to build a general model for KD. All the methods and techniques in KD can be summarized through the proposed model. By utilizing the proposed model, different methods in KD are better investigated and explored. The advantages and disadvantages of different approaches in KD can be better understood and develop a new strategy for KD can be possible. Using the proposed model, different KD methods are represented in an abstract view.