# Distilling Knowledge from Deep Networks with Applications to Healthcare Domain

arXiv: https://arxiv.org/abs/1512.03542 · HTML: https://mlsanity.com/paper/1512.03542/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2015.

## Built on by (most cited first, 16 of 16)

- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [Proto2Proto: Can you recognize the car, the way I do?](https://mlsanity.com/paper/2204.11830/) (2022)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [Accurate Explanation Model for Image Classifiers using Class Association Embedding](https://mlsanity.com/paper/2406.07961/) (2024)
- [Neural-to-Tree Policy Distillation with Policy Improvement Criterion](https://mlsanity.com/paper/2108.06898/) (2021)
- [Towards Interpretability of Speech Pause in Dementia Detection using Adversarial Learning](https://mlsanity.com/paper/2111.07454/) (2021)
- [Unsupervised Discovery of Clinical Disease Signatures Using Probabilistic Independence](https://mlsanity.com/paper/2402.05802/) (2024)
- [Absolute Value Constraint: The Reason for Invalid Performance Evaluation Results of Neural Network Models for Stock Price Prediction](https://mlsanity.com/paper/2101.10942/) (2021)
- [RuDi: Explaining Behavior Sequence Models by Automatic Statistics Generation and Rule Distillation](https://mlsanity.com/paper/2208.07211/) (2022)
- [Predicting Visit Cost of Obstructive Sleep Apnea using Electronic Healthcare Records with Transformer](https://mlsanity.com/paper/2301.12289/) (2023)
- [CSTRL: Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarization](https://mlsanity.com/paper/2503.05750/) (2025)
- [Explainable AI the Latest Advancements and New Trends](https://mlsanity.com/paper/2505.07005/) (2025)
- [Many-to-One Knowledge Distillation of Real-Time Epileptic Seizure Detection for Low-Power Wearable Internet of Things Systems](https://mlsanity.com/paper/2208.00885/) (2022)
- [Active Globally Explainable Learning for Medical Images via Class Association Embedding and Cyclic Adversarial Generation](https://mlsanity.com/paper/2306.07306/) (2023)
- [Cryptogenic stroke and migraine: using probabilistic independence and machine learning to uncover latent sources of disease from the electronic health record](https://mlsanity.com/paper/2505.04631/) (2025)

## Abstract (arXiv)

Exponential growth in Electronic Healthcare Records (EHR) has resulted in new opportunities and urgent needs for discovery of meaningful data-driven representations and patterns of diseases in Computational Phenotyping research. Deep Learning models have shown superior performance for robust prediction in computational phenotyping tasks, but suffer from the issue of model interpretability which is crucial for clinicians involved in decision-making. In this paper, we introduce a novel knowledge-distillation approach called Interpretable Mimic Learning, to learn interpretable phenotype features for making robust prediction while mimicking the performance of deep learning models. Our framework uses Gradient Boosting Trees to learn interpretable features from deep learning models such as Stacked Denoising Autoencoder and Long Short-Term Memory. Exhaustive experiments on a real-world clinical time-series dataset show that our method obtains similar or better performance than the deep learning models, and it provides interpretable phenotypes for clinical decision making.