# Student-friendly Knowledge Distillation arXiv: https://arxiv.org/abs/2305.10893 ยท HTML: https://mlsanity.com/paper/2305.10893/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 53 of its 62 references that are in the map. - First posted on arXiv 18 May 2023. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 53) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016) - [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017) - [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017) - [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016) - [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016) - [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017) - [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017) - [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022) ## Built on by (most cited first, 11 of 11) - [Dynamic Temperature Knowledge Distillation](https://mlsanity.com/paper/2404.12711/) (2024) - [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025) - [Attention-guided Feature Distillation for Semantic Segmentation](https://mlsanity.com/paper/2403.05451/) (2024) - [Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching](https://mlsanity.com/paper/2410.06561/) (2024) - [Temperature-Driven Robust Disease Detection in Brain and Gastrointestinal Disorders via Context-Aware Adaptive Knowledge Distillation](https://mlsanity.com/paper/2505.06381/) (2025) - [AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectories](https://mlsanity.com/paper/2310.10541/) (2023) - [Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher](https://mlsanity.com/paper/2410.04140/) (2024) - [PLD: A Choice-Theoretic List-Wise Knowledge Distillation](https://mlsanity.com/paper/2506.12542/) (2025) - [REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency](https://mlsanity.com/paper/2602.04677/) (2026) - [LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering](https://mlsanity.com/paper/2605.09384/) (2026) - [Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it](https://mlsanity.com/paper/2608.23752/) (2026) ## Abstract (arXiv) In knowledge distillation, the knowledge from the teacher model is often too complex for the student model to thoroughly process. However, good teachers in real life always simplify complex material before teaching it to students. Inspired by this fact, we propose student-friendly knowledge distillation (SKD) to simplify teacher output into new knowledge representations, which makes the learning of the student model easier and more effective. SKD contains a softening processing and a learning simplifier. First, the softening processing uses the temperature hyperparameter to soften the output logits of the teacher model, which simplifies the output to some extent and makes it easier for the learning simplifier to process. The learning simplifier utilizes the attention mechanism to further simplify the knowledge of the teacher model and is jointly trained with the student model using the distillation loss, which means that the process of simplification is correlated with the training objective of the student model and ensures that the simplified new teacher knowledge representation is more suitable for the specific student model. Furthermore, since SKD does not change the form of the distillation loss, it can be easily combined with other distillation methods that are based on the logits or features of intermediate layers to enhance its effectiveness. Therefore, SKD has wide applicability. The experimental results on the CIFAR-100 and ImageNet datasets show that our method achieves state-of-the-art performance while maintaining high training efficiency.