# Hierarchical interpretations for neural network predictions

arXiv: https://arxiv.org/abs/1806.05337 · HTML: https://mlsanity.com/paper/1806.05337/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 30 of 66)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Faith-Shap: The Faithful Shapley Interaction Index](https://mlsanity.com/paper/2203.00870/) (2022)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [On Guaranteed Optimal Robust Explanations for NLP Models](https://mlsanity.com/paper/2105.03640/) (2021)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [SHAP-IQ: Unified Approximation of any-order Shapley Interactions](https://mlsanity.com/paper/2303.01179/) (2023)
- [Discretized Integrated Gradients for Explaining Language Models](https://mlsanity.com/paper/2108.13654/) (2021)
- [SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers](https://mlsanity.com/paper/2103.12279/) (2021)
- [Shapley Explanation Networks](https://mlsanity.com/paper/2104.02297/) (2021)
- [Finding and Fixing Spurious Patterns with Explanations](https://mlsanity.com/paper/2106.02112/) (2021)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [Fast Axiomatic Attribution for Neural Networks](https://mlsanity.com/paper/2111.07668/) (2021)
- [Adaptive wavelet distillation from neural networks through interpretations](https://mlsanity.com/paper/2107.09145/) (2021)
- [Towards Robust Classification Model by Counterfactual and Invariant Data Generation](https://mlsanity.com/paper/2106.01127/) (2021)
- [Does a Neural Network Really Encode Symbolic Concepts?](https://mlsanity.com/paper/2302.13080/) (2023)
- [Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces](https://mlsanity.com/paper/2212.14855/) (2022)
- [Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition](https://mlsanity.com/paper/2407.00886/) (2024)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [DEGREE: Decomposition Based Explanation For Graph Neural Networks](https://mlsanity.com/paper/2305.12895/) (2023)
- [A Unified Game-Theoretic Interpretation of Adversarial Robustness](https://mlsanity.com/paper/2111.03536/) (2021)
- [Fast Hierarchical Games for Image Explanations](https://mlsanity.com/paper/2104.06164/) (2021)
- [Interpreting Attributions and Interactions of Adversarial Attacks](https://mlsanity.com/paper/2108.06895/) (2021)
- [Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks](https://mlsanity.com/paper/2104.04488/) (2021)
- [Multi-Level Explanations for Generative Language Models](https://mlsanity.com/paper/2403.14459/) (2024)
- [Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?](https://mlsanity.com/paper/2105.10719/) (2021)
- [3x2: 3D Object Part Segmentation by 2D Semantic Correspondences](https://mlsanity.com/paper/2407.09648/) (2024)
- [Let the CAT out of the bag: Contrastive Attributed explanations for Text](https://mlsanity.com/paper/2109.07983/) (2021)
- [Feature Interactions Reveal Linguistic Structure in Language Models](https://mlsanity.com/paper/2306.12181/) (2023)
- [Improving Interpretability via Explicit Word Interaction Graph Layer](https://mlsanity.com/paper/2302.02016/) (2023)

## Abstract (arXiv)

Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method, agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. Using examples from Stanford Sentiment Treebank and ImageNet, we show that ACD is effective at diagnosing incorrect predictions and identifying dataset bias. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.