# Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classification

arXiv: https://arxiv.org/abs/2307.04343 · HTML: https://mlsanity.com/paper/2307.04343/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 71 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients](https://mlsanity.com/paper/1711.09404/) (2017)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023)
- [Identifying and Controlling Important Neurons in Neural Machine Translation](https://mlsanity.com/paper/1811.01157/) (2018)
- [Interpreting Deep Visual Representations via Network Dissection](https://mlsanity.com/paper/1711.05611/) (2017)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT](https://mlsanity.com/paper/2304.08448/) (2023)
- [When Brain-inspired AI Meets AGI](https://mlsanity.com/paper/2303.15935/) (2023)
- [Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics](https://mlsanity.com/paper/2304.01938/) (2023)
- [Natural Neural Networks](https://mlsanity.com/paper/1507.00210/) (2015)

## Built on by (most cited first, 5 of 5)

- [Holistic Evaluation of GPT-4V for Biomedical Imaging](https://mlsanity.com/paper/2312.05256/) (2023)
- [RadOnc-GPT: A Large Language Model for Radiation Oncology](https://mlsanity.com/paper/2309.10160/) (2023)
- [Mass Concept Erasure in Diffusion Models with Concept Hierarchy](https://mlsanity.com/paper/2601.03305/) (2026)
- [PharmacyGPT: The AI Pharmacist](https://mlsanity.com/paper/2307.10432/) (2023)
- [What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits](https://mlsanity.com/paper/2409.02335/) (2024)

## Abstract (arXiv)

With the popularity of deep neural networks (DNNs), model interpretability is becoming a critical concern. Many approaches have been developed to tackle the problem through post-hoc analysis, such as explaining how predictions are made or understanding the meaning of neurons in middle layers. Nevertheless, these methods can only discover the patterns or rules that naturally exist in models. In this work, rather than relying on post-hoc schemes, we proactively instill knowledge to alter the representation of human-understandable concepts in hidden layers. Specifically, we use a hierarchical tree of semantic concepts to store the knowledge, which is leveraged to regularize the representations of image data instances while training deep models. The axes of the latent space are aligned with the semantic concepts, where the hierarchical relations between concepts are also preserved. Experiments on real-world image datasets show that our method improves model interpretability, showing better disentanglement of semantic concepts, without negatively affecting model classification performance.