# Approximation Trees: Statistical Stability in Model Distillation

arXiv: https://arxiv.org/abs/1808.07573 · HTML: https://mlsanity.com/paper/1808.07573/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2018.

## Built on by (most cited first, 9 of 9)

- [S-LIME: Stabilized-LIME for Model Explanation](https://mlsanity.com/paper/2106.07875/) (2021)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [How to Evaluate Uncertainty Estimates in Machine Learning for Regression?](https://mlsanity.com/paper/2106.03395/) (2021)
- [Towards a theory of model distillation](https://mlsanity.com/paper/2403.09053/) (2024)
- [Stabilizing Estimates of Shapley Values with Control Variates](https://mlsanity.com/paper/2310.07672/) (2023)
- [Marginal Effects for Non-Linear Prediction Functions](https://mlsanity.com/paper/2201.08837/) (2022)
- [A Generic Approach for Reproducible Model Distillation](https://mlsanity.com/paper/2211.12631/) (2022)
- [WildWood: a new Random Forest algorithm](https://mlsanity.com/paper/2109.08010/) (2021)
- [MindfulLIME: A Stable Solution for Explanations of Machine Learning Models with Enhanced Localization Precision -- A Medical Image Case Study](https://mlsanity.com/paper/2503.20758/) (2025)

## Abstract (arXiv)

This paper examines the stability of learned explanations for black-box predictions via model distillation with decision trees. One approach to intelligibility in machine learning is to use an understandable `student' model to mimic the output of an accurate `teacher'. Here, we consider the use of regression trees as a student model, in which nodes of the tree can be used as `explanations' for particular predictions, and the whole structure of the tree can be used as a global representation of the resulting function. However, individual trees are sensitive to the particular data sets used to train them, and an interpretation of a student model may be suspect if small changes in the training data have a large effect on it. In this context, access to outcomes from a teacher helps to stabilize the greedy splitting strategy by generating a much larger corpus of training examples than was originally available. We develop tests to ensure that enough examples are generated at each split so that the same splitting rule would be chosen with high probability were the tree to be re trained. Further, we develop a stopping rule to indicate how deep the tree should be built based on recent results on the variability of Random Forests when these are used as the teacher. We provide concrete examples of these procedures on the CAD-MDD and COMPAS data sets.