# Improving the Interpretability of Deep Neural Networks with Knowledge Distillation

arXiv: https://arxiv.org/abs/1812.10924 · HTML: https://mlsanity.com/paper/1812.10924/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2018.

## Built on by (most cited first, 30 of 38)

- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Incorporating Domain Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2103.00180/) (2021)
- [On Guaranteed Optimal Robust Explanations for NLP Models](https://mlsanity.com/paper/2105.03640/) (2021)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Label Poisoning is All You Need](https://mlsanity.com/paper/2310.18933/) (2023)
- [Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations](https://mlsanity.com/paper/2112.09669/) (2021)
- [Proto2Proto: Can you recognize the car, the way I do?](https://mlsanity.com/paper/2204.11830/) (2022)
- [Online Continual Learning with Contrastive Vision Transformer](https://mlsanity.com/paper/2207.13516/) (2022)
- [Evaluating Feature Attribution Methods in the Image Domain](https://mlsanity.com/paper/2202.12270/) (2022)
- [Understanding the Gains from Repeated Self-Distillation](https://mlsanity.com/paper/2407.04600/) (2024)
- [Digital-SC: Digital Semantic Communication with Adaptive Network Split and Learned Non-Linear Quantization](https://mlsanity.com/paper/2305.13553/) (2023)
- [Causality-Inspired Taxonomy for Explainable Artificial Intelligence](https://mlsanity.com/paper/2208.09500/) (2022)
- [Deeply Explain CNN via Hierarchical Decomposition](https://mlsanity.com/paper/2201.09205/) (2022)
- [Accurate Explanation Model for Image Classifiers using Class Association Embedding](https://mlsanity.com/paper/2406.07961/) (2024)
- [Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator](https://mlsanity.com/paper/2407.11078/) (2024)
- [AI Readiness in Healthcare through Storytelling XAI](https://mlsanity.com/paper/2410.18725/) (2024)
- [Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs](https://mlsanity.com/paper/2207.02295/) (2022)
- [Variance Tolerance Factors For Interpreting ALL Neural Networks](https://mlsanity.com/paper/2209.13858/) (2022)
- [Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models](https://mlsanity.com/paper/2604.13332/) (2026)
- [Model Compression for Dynamic Forecast Combination](https://mlsanity.com/paper/2104.01830/) (2021)
- [How to Explain Neural Networks: an Approximation Perspective](https://mlsanity.com/paper/2105.07831/) (2021)
- [Balanced background and explanation data are needed in explaining deep learning models with SHAP: An empirical study on clinical decision making](https://mlsanity.com/paper/2206.04050/) (2022)
- [Explaining Neural Networks without Access to Training Data](https://mlsanity.com/paper/2206.04891/) (2022)
- [RuDi: Explaining Behavior Sequence Models by Automatic Statistics Generation and Rule Distillation](https://mlsanity.com/paper/2208.07211/) (2022)
- [Distilling Deep RL Models Into Interpretable Neuro-Fuzzy Systems](https://mlsanity.com/paper/2209.03357/) (2022)
- [Towards Explaining Autonomy with Verbalised Decision Tree States](https://mlsanity.com/paper/2209.13985/) (2022)
- [Using Knowledge Distillation to improve interpretable models in a retail banking context](https://mlsanity.com/paper/2209.15496/) (2022)
- [Rule-Extraction Methods From Feedforward Neural Networks: A Systematic Literature Review](https://mlsanity.com/paper/2312.12878/) (2023)
- [A Human-In-The-Loop Approach for Improving Fairness in Predictive Business Process Monitoring](https://mlsanity.com/paper/2508.17477/) (2025)
- [A Review of Some Techniques for Inclusion of Domain-Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2107.10295/) (2021)

## Abstract (arXiv)

Deep Neural Networks have achieved huge success at a wide spectrum of applications from language modeling, computer vision to speech recognition. However, nowadays, good performance alone is not sufficient to satisfy the needs of practical deployment where interpretability is demanded for cases involving ethics and mission critical applications. The complex models of Deep Neural Networks make it hard to understand and reason the predictions, which hinders its further progress. To tackle this problem, we apply the Knowledge Distillation technique to distill Deep Neural Networks into decision trees in order to attain good performance and interpretability simultaneously. We formulate the problem at hand as a multi-output regression problem and the experiments demonstrate that the student model achieves significantly better accuracy performance (about 1% to 5%) than vanilla decision trees at the same level of tree depth. The experiments are implemented on the TensorFlow platform to make it scalable to big datasets. To the best of our knowledge, we are the first to distill Deep Neural Networks into vanilla decision trees on multi-class datasets.