# Learning Interpretation with Explainable Knowledge Distillation

arXiv: https://arxiv.org/abs/2111.06945 · HTML: https://mlsanity.com/paper/2111.06945/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 18 of 18)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement](https://mlsanity.com/paper/1511.03995/) (2015)
- [PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks](https://mlsanity.com/paper/2010.05788/) (2020)
- [DEVDAN: Deep Evolving Denoising Autoencoder](https://mlsanity.com/paper/1910.04062/) (2019)

## Built on by (most cited first, 13 of 13)

- [Mind the gap: Challenges of deep learning approaches to Theory of Mind](https://mlsanity.com/paper/2203.16540/) (2022)
- [Good Teachers Explain: Explanation-Enhanced Knowledge Distillation](https://mlsanity.com/paper/2402.03119/) (2024)
- [Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat](https://mlsanity.com/paper/2307.05350/) (2023)
- [Causality-Inspired Taxonomy for Explainable Artificial Intelligence](https://mlsanity.com/paper/2208.09500/) (2022)
- [Interpretability-Aware Vision Transformer](https://mlsanity.com/paper/2309.08035/) (2023)
- [On the Generalization vs Fidelity Paradox in Knowledge Distillation](https://mlsanity.com/paper/2505.15442/) (2025)
- [Toward Multiple Specialty Learners for Explaining GNNs via Online Knowledge Distillation](https://mlsanity.com/paper/2210.11094/) (2022)
- [Exploring the Lottery Ticket Hypothesis with Explainability Methods: Insights into Sparse Network Performance](https://mlsanity.com/paper/2307.13698/) (2023)
- [Learning to Reason: Temporal Saliency Distillation for Interpretable Knowledge Transfer](https://mlsanity.com/paper/2601.04263/) (2026)
- [PGX: A Multi-level GNN Explanation Framework Based on Separate Knowledge Distillation Processes](https://mlsanity.com/paper/2208.03075/) (2022)
- [XD-RCDepth: Lightweight Radar-Camera Depth Estimation with Explainability-Aligned and Distribution-Aware Distillation](https://mlsanity.com/paper/2510.13565/) (2025)
- [What's meant by explainable model: A Scoping Review](https://mlsanity.com/paper/2307.09673/) (2023)
- [LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models](https://mlsanity.com/paper/2601.22928/) (2026)

## Abstract (arXiv)

Knowledge Distillation (KD) has been considered as a key solution in model compression and acceleration in recent years. In KD, a small student model is generally trained from a large teacher model by minimizing the divergence between the probabilistic outputs of the two. However, as demonstrated in our experiments, existing KD methods might not transfer critical explainable knowledge of the teacher to the student, i.e. the explanations of predictions made by the two models are not consistent. In this paper, we propose a novel explainable knowledge distillation model, called XDistillation, through which both the performance the explanations' information are transferred from the teacher model to the student model. The XDistillation model leverages the idea of convolutional autoencoders to approximate the teacher explanations. Our experiments shows that models trained by XDistillation outperform those trained by conventional KD methods not only in term of predictive accuracy but also faithfulness to the teacher models.