# Good Teachers Explain: Explanation-Enhanced Knowledge Distillation arXiv: https://arxiv.org/abs/2402.03119 ยท HTML: https://mlsanity.com/paper/2402.03119/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 45 of its 57 references that are in the map. - First posted on arXiv 5 Feb 2024. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 45) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019) - [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022) - [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015) - [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019) - [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019) - [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022) - [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021) - [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017) - [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021) - [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021) - [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021) - [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021) - [B-cos Networks: Alignment is All We Need for Interpretability](https://mlsanity.com/paper/2205.10268/) (2022) ## Built on by (most cited first, 14 of 14) - [B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable](https://mlsanity.com/paper/2411.00715/) (2024) - [Cross-Architecture Distillation Made Simple with Redundancy Suppression](https://mlsanity.com/paper/2507.21844/) (2025) - [Mitigating Negative Flips via Margin Preserving Training](https://mlsanity.com/paper/2511.08322/) (2025) - [OMENN: One Matrix to Explain Neural Networks](https://mlsanity.com/paper/2412.02399/) (2024) - [Learning to Reason: Temporal Saliency Distillation for Interpretable Knowledge Transfer](https://mlsanity.com/paper/2601.04263/) (2026) - [Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspaces](https://mlsanity.com/paper/2601.05913/) (2026) - [XD-RCDepth: Lightweight Radar-Camera Depth Estimation with Explainability-Aligned and Distribution-Aware Distillation](https://mlsanity.com/paper/2510.13565/) (2025) - [FaCT: Faithful Concept Traces for Explaining Neural Network Decisions](https://mlsanity.com/paper/2510.25512/) (2025) - [Investigating the Robustness of Subtask Distillation under Spurious Correlation](https://mlsanity.com/paper/2602.00852/) (2026) - [Training with Explanations Alone: A New Paradigm to Prevent Shortcut Learning](https://mlsanity.com/paper/2407.09788/) (2024) - [A Closer Look at Knowledge Distillation in Spiking Neural Network Training](https://mlsanity.com/paper/2511.06902/) (2025) - [Hybrid Attribution Priors for Explainable and Robust Model Training](https://mlsanity.com/paper/2512.14719/) (2025) - [Generative Diffusion Prior Distillation for Long-Context Knowledge Transfer](https://mlsanity.com/paper/2605.11414/) (2026) - [Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it](https://mlsanity.com/paper/2608.23752/) (2026) ## Abstract (arXiv) Knowledge Distillation (KD) has proven effective for compressing large teacher models into smaller student models. While it is well known that student models can achieve similar accuracies as the teachers, it has also been shown that they nonetheless often do not learn the same function. It is, however, often highly desirable that the student's and teacher's functions share similar properties such as basing the prediction on the same input features, as this ensures that students learn the 'right features' from the teachers. In this work, we explore whether this can be achieved by not only optimizing the classic KD loss but also the similarity of the explanations generated by the teacher and the student. Despite the idea being simple and intuitive, we find that our proposed 'explanation-enhanced' KD (e2KD) (1) consistently provides large gains in terms of accuracy and student-teacher agreement, (2) ensures that the student learns from the teacher to be right for the right reasons and to give similar explanations, and (3) is robust with respect to the model architectures, the amount of training data, and even works with 'approximate', pre-computed explanations.