# DynamicKD: An Effective Knowledge Distillation via Dynamic Entropy Correction-Based Distillation for Gap Optimizing arXiv: https://arxiv.org/abs/2305.05233 ยท HTML: https://mlsanity.com/paper/2305.05233/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 40 of its 68 references that are in the map. - First posted on arXiv 9 May 2023. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 40) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019) - [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019) - [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019) - [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014) - [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017) - [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021) - [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019) - [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018) - [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018) - [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018) - [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018) - [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020) - [Domain Adaptation for Semantic Segmentation with Maximum Squares Loss](https://mlsanity.com/paper/1909.13589/) (2019) - [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021) - [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020) ## Built on by (most cited first, 7 of 7) - [EA-KD: Entropy-based Adaptive Knowledge Distillation](https://mlsanity.com/paper/2311.13621/) (2023) - [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024) - [Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias](https://mlsanity.com/paper/2412.09874/) (2024) - [EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer](https://mlsanity.com/paper/2605.01732/) (2026) - [Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher](https://mlsanity.com/paper/2410.04140/) (2024) - [Hierarchical mutual distillation for multi-view fusion: Learning from all possible view combinations](https://mlsanity.com/paper/2411.10077/) (2024) - [SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines](https://mlsanity.com/paper/2601.01484/) (2026) ## Abstract (arXiv) The knowledge distillation uses a high-performance teacher network to guide the student network. However, the performance gap between the teacher and student networks can affect the student's training. This paper proposes a novel knowledge distillation algorithm based on dynamic entropy correction to reduce the gap by adjusting the student instead of the teacher. Firstly, the effect of changing the output entropy (short for output information entropy) in the student on the distillation loss is analyzed in theory. This paper shows that correcting the output entropy can reduce the gap. Then, a knowledge distillation algorithm based on dynamic entropy correction is created, which can correct the output entropy in real-time with an entropy controller updated dynamically by the distillation loss. The proposed algorithm is validated on the CIFAR100 and ImageNet. The comparison with various state-of-the-art distillation algorithms shows impressive results, especially in the experiment on the CIFAR100 regarding teacher-student pair resnet32x4-resnet8x4. The proposed algorithm raises 2.64 points over the traditional distillation algorithm and 0.87 points over the state-of-the-art algorithm CRD in classification accuracy, demonstrating its effectiveness and efficiency.