# Knowledge Distillation of Convolutional Neural Networks through Feature Map Transformation using Decision Trees

arXiv: https://arxiv.org/abs/2403.06089 · HTML: https://mlsanity.com/paper/2403.06089/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 5 of its 6 references that are in the map.
- First posted on arXiv 10 Mar 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 5 of 5)

- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Adaptive Neural Trees](https://mlsanity.com/paper/1807.06699/) (2018)
- [Improving the Interpretability of Deep Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/1812.10924/) (2018)
- [Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness](https://mlsanity.com/paper/1903.05965/) (2019)

## Abstract (arXiv)

The interpretation of reasoning by Deep Neural Networks (DNN) is still challenging due to their perceived black-box nature. Therefore, deploying DNNs in several real-world tasks is restricted by the lack of transparency of these models. We propose a distillation approach by extracting features from the final layer of the convolutional neural network (CNN) to address insights to its reasoning. The feature maps in the final layer of a CNN are transformed into a one-dimensional feature vector using a fully connected layer. Subsequently, the extracted features are used to train a decision tree to achieve the best accuracy under constraints of depth and nodes. We use the medical images of dermaMNIST, octMNIST, and pneumoniaMNIST from the medical MNIST datasets to demonstrate our proposed work. We observed that performance of the decision tree is as good as a CNN with minimum complexity. The results encourage interpreting decisions made by the CNNs using decision trees.