# A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

arXiv: https://arxiv.org/abs/2105.00290 · HTML: https://mlsanity.com/paper/2105.00290/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks](https://mlsanity.com/paper/1810.02244/) (2018)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)

## Built on by (most cited first, 30 of 33)

- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [Diffusion Models for Counterfactual Explanations](https://mlsanity.com/paper/2203.15636/) (2022)
- [Learning Bottleneck Concepts in Image Classification](https://mlsanity.com/paper/2304.10131/) (2023)
- [Adversarial Counterfactual Visual Explanations](https://mlsanity.com/paper/2303.09962/) (2023)
- [Revisiting Open World Object Detection](https://mlsanity.com/paper/2201.00471/) (2022)
- [Text-to-Image Models for Counterfactual Explanations: a Black-Box Approach](https://mlsanity.com/paper/2309.07944/) (2023)
- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [Contributions of Shape, Texture, and Color in Visual Recognition](https://mlsanity.com/paper/2207.09510/) (2022)
- [ECLAD: Extracting Concepts with Local Aggregated Descriptors](https://mlsanity.com/paper/2206.04531/) (2022)
- [Spatial-temporal Concept based Explanation of 3D ConvNets](https://mlsanity.com/paper/2206.05275/) (2022)
- [Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention](https://mlsanity.com/paper/2204.04601/) (2022)
- [Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability](https://mlsanity.com/paper/2304.14864/) (2023)
- [Graph-Based Similarity of Neural Network Representations](https://mlsanity.com/paper/2111.11165/) (2021)
- [Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces](https://mlsanity.com/paper/2311.14435/) (2023)
- [Object Detector Differences when using Synthetic and Real Training Data](https://mlsanity.com/paper/2312.00694/) (2023)
- [Revealing Similar Semantics Inside CNNs: An Interpretable Concept-based Comparison of Feature Spaces](https://mlsanity.com/paper/2305.07663/) (2023)
- [Learning Transferable Conceptual Prototypes for Interpretable Unsupervised Domain Adaptation](https://mlsanity.com/paper/2310.08071/) (2023)
- [Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders](https://mlsanity.com/paper/2506.23951/) (2025)
- [Scale-Preserving Automatic Concept Extraction (SPACE)](https://mlsanity.com/paper/2308.06022/) (2023)
- [Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach](https://mlsanity.com/paper/2411.13302/) (2024)
- [Schema Inference for Interpretable Image Classification](https://mlsanity.com/paper/2303.06635/) (2023)
- [On Model Explanations with Transferable Neural Pathways](https://mlsanity.com/paper/2309.09887/) (2023)
- [Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing](https://mlsanity.com/paper/2312.07875/) (2023)
- [The two-way knowledge interaction interface between humans and neural networks](https://mlsanity.com/paper/2401.05461/) (2024)
- [Disentangling Visual Transformers: Patch-level Interpretability for Image Classification](https://mlsanity.com/paper/2502.17196/) (2025)
- [Counterfactual Visual Explanation via Causally-Guided Adversarial Steering](https://mlsanity.com/paper/2507.09881/) (2025)
- [A Cognitive Explainer for Fetal ultrasound images classifier Based on Medical Concepts](https://mlsanity.com/paper/2201.07798/) (2022)
- [Knowledge-Aware Neuron Interpretation for Scene Classification](https://mlsanity.com/paper/2401.15820/) (2024)
- [The Anatomy of Adversarial Attacks: Concept-based XAI Dissection](https://mlsanity.com/paper/2403.16782/) (2024)

## Abstract (arXiv)

Despite substantial progress in applying neural networks (NN) to a wide variety of areas, they still largely suffer from a lack of transparency and interpretability. While recent developments in explainable artificial intelligence attempt to bridge this gap (e.g., by visualizing the correlation between input pixels and final outputs), these approaches are limited to explaining low-level relationships, and crucially, do not provide insights on error correction. In this work, we propose a framework (VRX) to interpret classification NNs with intuitive structural visual concepts. Given a trained classification model, the proposed VRX extracts relevant class-specific visual concepts and organizes them using structural concept graphs (SCG) based on pairwise concept relationships. By means of knowledge distillation, we show VRX can take a step towards mimicking the reasoning process of NNs and provide logical, concept-level explanations for final model decisions. With extensive experiments, we empirically show VRX can meaningfully answer "why" and "why not" questions about the prediction, providing easy-to-understand insights about the reasoning process. We also show that these insights can potentially provide guidance on improving NN's performance.