# A Survey on Recent Teacher-student Learning Studies

arXiv: https://arxiv.org/abs/2304.04615 · HTML: https://mlsanity.com/paper/2304.04615/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 15 of 15)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation](https://mlsanity.com/paper/2202.03680/) (2022)
- [Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2112.08996/) (2021)
- [DisWOT: Student Architecture Search for Distillation WithOut Training](https://mlsanity.com/paper/2303.15678/) (2023)
- [Matching Guided Distillation](https://mlsanity.com/paper/2008.09958/) (2020)
- [Prior-Guided One-shot Neural Architecture Search](https://mlsanity.com/paper/2206.13329/) (2022)
- [GP-NAS-ensemble: a model for NAS Performance Prediction](https://mlsanity.com/paper/2301.09231/) (2023)
- [RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies](https://mlsanity.com/paper/2301.09850/) (2023)
- [Progressive Meta-Pooling Learning for Lightweight Image Classification Model](https://mlsanity.com/paper/2301.10038/) (2023)

## Built on by (most cited first, 1 of 1)

- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)

## Abstract (arXiv)

Knowledge distillation is a method of transferring the knowledge from a complex deep neural network (DNN) to a smaller and faster DNN, while preserving its accuracy. Recent variants of knowledge distillation include teaching assistant distillation, curriculum distillation, mask distillation, and decoupling distillation, which aim to improve the performance of knowledge distillation by introducing additional components or by changing the learning process. Teaching assistant distillation involves an intermediate model called the teaching assistant, while curriculum distillation follows a curriculum similar to human education. Mask distillation focuses on transferring the attention mechanism learned by the teacher, and decoupling distillation decouples the distillation loss from the task loss. Overall, these variants of knowledge distillation have shown promising results in improving the performance of knowledge distillation.