# Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness

arXiv: https://arxiv.org/abs/1903.05965 · HTML: https://mlsanity.com/paper/1903.05965/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2019.

## Built on by (most cited first, 4 of 4)

- [Fast Sparse Decision Tree Optimization via Reference Ensembles](https://mlsanity.com/paper/2112.00798/) (2021)
- [Enhancing Pre-trained Representation Classifiability can Boost its Interpretability](https://mlsanity.com/paper/2510.24105/) (2025)
- [Using Knowledge Distillation to improve interpretable models in a retail banking context](https://mlsanity.com/paper/2209.15496/) (2022)
- [Knowledge Distillation of Convolutional Neural Networks through Feature Map Transformation using Decision Trees](https://mlsanity.com/paper/2403.06089/) (2024)

## Abstract (arXiv)

How to obtain a model with good interpretability and performance has always been an important research topic. In this paper, we propose rectified decision trees (ReDT), a knowledge distillation based decision trees rectification with high interpretability, small model size, and empirical soundness. Specifically, we extend the impurity calculation and the pure ending condition of the classical decision tree to propose a decision tree extension that allows the use of soft labels generated by a well-trained teacher model in training and prediction process. It is worth noting that for the acquisition of soft labels, we propose a new multiple cross-validation based method to reduce the effects of randomness and overfitting. These approaches ensure that ReDT retains excellent interpretability and even achieves fewer nodes than the decision tree in the aspect of compression while having relatively good performance. Besides, in contrast to traditional knowledge distillation, back propagation of the student model is not necessarily required in ReDT, which is an attempt of a new knowledge distillation approach. Extensive experiments are conducted, which demonstrates the superiority of ReDT in interpretability, compression, and empirical soundness.