# A Generic Approach for Reproducible Model Distillation

arXiv: https://arxiv.org/abs/2211.12631 · HTML: https://mlsanity.com/paper/2211.12631/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation](https://mlsanity.com/paper/1710.06169/) (2017)
- [Falling Rule Lists](https://mlsanity.com/paper/1411.5899/) (2014)
- [S-LIME: Stabilized-LIME for Model Explanation](https://mlsanity.com/paper/2106.07875/) (2021)
- [Applying Genetic Programming to Improve Interpretability in Machine Learning Models](https://mlsanity.com/paper/2005.09512/) (2020)
- [Approximation Trees: Statistical Stability in Model Distillation](https://mlsanity.com/paper/1808.07573/) (2018)
- [The Infinitesimal Jackknife and Combinations of Models](https://mlsanity.com/paper/2209.00147/) (2022)

## Built on by (most cited first, 3 of 3)

- [Towards a theory of model distillation](https://mlsanity.com/paper/2403.09053/) (2024)
- [Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method](https://mlsanity.com/paper/2508.14783/) (2025)
- [Towards Performatively Stable Equilibria in Decision-Dependent Games for Arbitrary Data Distribution Maps](https://mlsanity.com/paper/2509.02619/) (2025)

## Abstract (arXiv)

Model distillation has been a popular method for producing interpretable machine learning. It uses an interpretable "student" model to mimic the predictions made by the black box "teacher" model. However, when the student model is sensitive to the variability of the data sets used for training even when keeping the teacher fixed, the corresponded interpretation is not reliable. Existing strategies stabilize model distillation by checking whether a large enough corpus of pseudo-data is generated to reliably reproduce student models, but methods to do so have so far been developed for a specific student model. In this paper, we develop a generic approach for stable model distillation based on central limit theorem for the average loss. We start with a collection of candidate student models and search for candidates that reasonably agree with the teacher. Then we construct a multiple testing framework to select a corpus size such that the consistent student model would be selected under different pseudo samples. We demonstrate the application of our proposed approach on three commonly used intelligible models: decision trees, falling rule lists and symbolic regression. Finally, we conduct simulation experiments on Mammographic Mass and Breast Cancer datasets and illustrate the testing procedure throughout a theoretical analysis with Markov process. The code is publicly available at https://github.com/yunzhe-zhou/GenericDistillation.