# One Explanation is Not Enough: Structured Attention Graphs for Image Classification

arXiv: https://arxiv.org/abs/2011.06733 · HTML: https://mlsanity.com/paper/2011.06733/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2020.

## Built on by (most cited first, 30 of 32)

- [HIVE: Evaluating the Human Interpretability of Visual Explanations](https://mlsanity.com/paper/2112.03184/) (2021)
- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [Learning Support and Trivial Prototypes for Interpretable Image Classification](https://mlsanity.com/paper/2301.04011/) (2023)
- [Evolving Interpretable Visual Classifiers with Large Language Models](https://mlsanity.com/paper/2404.09941/) (2024)
- [Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods](https://mlsanity.com/paper/2212.06872/) (2022)
- [Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognition](https://mlsanity.com/paper/2312.00092/) (2023)
- [Causal Explanations for Image Classifiers](https://mlsanity.com/paper/2411.08875/) (2024)
- [Explaining Image Classifiers](https://mlsanity.com/paper/2401.13752/) (2024)
- [A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculture](https://mlsanity.com/paper/2210.11318/) (2022)
- [How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps](https://mlsanity.com/paper/2404.16042/) (2024)
- [Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracy](https://mlsanity.com/paper/2404.05238/) (2024)
- [Multiple Different Black Box Explanations for Image Classifiers](https://mlsanity.com/paper/2309.14309/) (2023)
- [Explaining Object Detectors via Collective Contribution of Pixels](https://mlsanity.com/paper/2412.00666/) (2024)
- [Free Argumentative Exchanges for Explaining Image Classifiers](https://mlsanity.com/paper/2502.12995/) (2025)
- [Attention IoU: Examining Biases in CelebA using Attention Maps](https://mlsanity.com/paper/2503.19846/) (2025)
- [I Am Big, You Are Little; I Am Right, You Are Wrong](https://mlsanity.com/paper/2507.23509/) (2025)
- [Care for the Mind Amid Chronic Diseases: An Interpretable AI Approach Using IoT](https://mlsanity.com/paper/2211.04509/) (2022)
- [Generic Attention-model Explainability by Weighted Relevance Accumulation](https://mlsanity.com/paper/2308.10240/) (2023)
- [You Only Explain Once](https://mlsanity.com/paper/2311.14081/) (2023)
- [Local-to-Global Logical Explanations for Deep Vision Models](https://mlsanity.com/paper/2601.13404/) (2026)
- [Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing](https://mlsanity.com/paper/2312.07875/) (2023)
- [How Video Meetings Change Your Expression](https://mlsanity.com/paper/2406.00955/) (2024)
- [Generating visual explanations from deep networks using implicit neural representations](https://mlsanity.com/paper/2501.11784/) (2025)
- [Interactivity x Explainability: Toward Understanding How Interactivity Can Improve Computer Vision Explanations](https://mlsanity.com/paper/2504.10745/) (2025)
- [PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution](https://mlsanity.com/paper/2606.18008/) (2026)
- [Comparing Importance Sampling Based Methods for Mitigating the Effect of Class Imbalance](https://mlsanity.com/paper/2402.18742/) (2024)
- [Learning local discrete features in explainable-by-design convolutional neural networks](https://mlsanity.com/paper/2411.00139/) (2024)
- [Evaluation of Black-Box XAI Approaches for Predictors of Values of Boolean Formulae](https://mlsanity.com/paper/2509.09982/) (2025)
- [Generating Part-Based Global Explanations Via Correspondence](https://mlsanity.com/paper/2509.15393/) (2025)

## Abstract (arXiv)

Attention maps are a popular way of explaining the decisions of convolutional networks for image classification. Typically, for each image of interest, a single attention map is produced, which assigns weights to pixels based on their importance to the classification. A single attention map, however, provides an incomplete understanding since there are often many other maps that explain a classification equally well. In this paper, we introduce structured attention graphs (SAGs), which compactly represent sets of attention maps for an image by capturing how different combinations of image regions impact a classifier's confidence. We propose an approach to compute SAGs and a visualization for SAGs so that deeper insight can be gained into a classifier's decisions. We conduct a user study comparing the use of SAGs to traditional attention maps for answering counterfactual questions about image classifications. Our results show that the users are more correct when answering comparative counterfactual questions based on SAGs compared to the baselines.