# Global Concept-Based Interpretability for Graph Neural Networks via Neuron Analysis

arXiv: https://arxiv.org/abs/2208.10609 · HTML: https://mlsanity.com/paper/2208.10609/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 26 of 26)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [GNNExplainer: Generating Explanations for Graph Neural Networks](https://mlsanity.com/paper/1903.03894/) (2019)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Parameterized Explainer for Graph Neural Network](https://mlsanity.com/paper/2011.04573/) (2020)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [XGNN: Towards Model-Level Explanations of Graph Neural Networks](https://mlsanity.com/paper/2006.02587/) (2020)
- [PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks](https://mlsanity.com/paper/2010.05788/) (2020)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability](https://mlsanity.com/paper/2204.08570/) (2022)
- [A Tutorial on Kernel Density Estimation and Recent Advances](https://mlsanity.com/paper/1704.03924/) (2017)
- [Explanation by Progressive Exaggeration](https://mlsanity.com/paper/1911.00483/) (2019)
- [DIG: A Turnkey Library for Diving into Graph Deep Learning Research](https://mlsanity.com/paper/2103.12608/) (2021)
- [Entropy-based Logic Explanations of Neural Networks](https://mlsanity.com/paper/2106.06804/) (2021)
- [Message passing all the way up](https://mlsanity.com/paper/2202.11097/) (2022)
- [GCExplainer: Human-in-the-Loop Concept-based Explanations for Graph Neural Networks](https://mlsanity.com/paper/2107.11889/) (2021)
- [Survey of explainable machine learning with visual and granular methods beyond quasi-explanations](https://mlsanity.com/paper/2009.10221/) (2020)
- [Algorithmic Concept-based Explainable Reasoning](https://mlsanity.com/paper/2107.07493/) (2021)

## Built on by (most cited first, 30 of 32)

- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Interpretable Neural-Symbolic Concept Reasoning](https://mlsanity.com/paper/2304.14068/) (2023)
- [A Survey on Explainability of Graph Neural Networks](https://mlsanity.com/paper/2306.01958/) (2023)
- [Semi-supervised Concept Bottleneck Models](https://mlsanity.com/paper/2406.18992/) (2024)
- [GNNX-BENCH: Unravelling the Utility of Perturbation-based GNN Explainers through In-depth Benchmarking](https://mlsanity.com/paper/2310.01794/) (2023)
- [View-based Explanations for Graph Neural Networks](https://mlsanity.com/paper/2401.02086/) (2024)
- [L2XGNN: Learning to Explain Graph Neural Networks](https://mlsanity.com/paper/2209.14402/) (2022)
- [MAGE: Model-Level Graph Neural Networks Explanations via Motif-based Graph Generation](https://mlsanity.com/paper/2405.12519/) (2024)
- [From Nodes to Narratives: Explaining Graph Neural Networks with LLMs and Graph Context](https://mlsanity.com/paper/2508.07117/) (2025)
- [Interpretable Graph Neural Networks for Tabular Data](https://mlsanity.com/paper/2308.08945/) (2023)
- [Robustness questions the interpretability of graph neural networks: what to do?](https://mlsanity.com/paper/2505.02566/) (2025)
- [Prediction via Shapley Value Regression](https://mlsanity.com/paper/2505.04775/) (2025)
- [It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Models](https://mlsanity.com/paper/2308.08268/) (2023)
- [Everybody Needs a Little HELP: Explaining Graphs via Hierarchical Concepts](https://mlsanity.com/paper/2311.15112/) (2023)
- [Automated Molecular Concept Generation and Labeling with Large Language Models](https://mlsanity.com/paper/2406.09612/) (2024)
- [A Benchmark for Fairness-Aware Graph Learning](https://mlsanity.com/paper/2407.12112/) (2024)
- [Superposition in Graph Neural Networks](https://mlsanity.com/paper/2509.00928/) (2025)
- [GnnXemplar: Exemplars to Explanations -- Natural Language Rules for Global GNN Interpretability](https://mlsanity.com/paper/2509.18376/) (2025)
- [Interpretable Graph Neural Networks for Heterogeneous Tabular Data](https://mlsanity.com/paper/2408.07661/) (2024)
- [Decompose the model: Mechanistic interpretability in image models with Generalized Integrated Gradients (GIG)](https://mlsanity.com/paper/2409.01610/) (2024)
- [Disentangled and Self-Explainable Node Representation Learning](https://mlsanity.com/paper/2410.21043/) (2024)
- [LogicXGNN: Grounded Logical Rules for Explaining Graph Neural Networks](https://mlsanity.com/paper/2503.19476/) (2025)
- [Interpretable Neuropsychiatric Diagnosis via Concept-Guided Graph Neural Networks](https://mlsanity.com/paper/2510.03351/) (2025)
- [Beyond Message Passing: A Symbolic Alternative for Expressive and Interpretable Graph Learning](https://mlsanity.com/paper/2602.16947/) (2026)
- [Empowering Counterfactual Reasoning over Graph Neural Networks through Inductivity](https://mlsanity.com/paper/2306.04835/) (2023)
- [Do graph neural network states contain graph properties?](https://mlsanity.com/paper/2411.02168/) (2024)
- [Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning](https://mlsanity.com/paper/2502.00684/) (2025)
- [Model-Level GNN Explanations via Rule-to-Graph Readout for Logit Reconstruction](https://mlsanity.com/paper/2503.09051/) (2025)
- [Framework GNN-AID: Graph Neural Network Analysis Interpretation and Defense](https://mlsanity.com/paper/2505.03424/) (2025)
- [Multimodal Attention-Aware Fusion for Diagnosing Distal Myopathy: Evaluating Model Interpretability and Clinician Trust](https://mlsanity.com/paper/2508.01316/) (2025)

## Abstract (arXiv)

Graph neural networks (GNNs) are highly effective on a variety of graph-related tasks; however, they lack interpretability and transparency. Current explainability approaches are typically local and treat GNNs as black-boxes. They do not look inside the model, inhibiting human trust in the model and explanations. Motivated by the ability of neurons to detect high-level semantic concepts in vision models, we perform a novel analysis on the behaviour of individual GNN neurons to answer questions about GNN interpretability, and propose new metrics for evaluating the interpretability of GNN neurons. We propose a novel approach for producing global explanations for GNNs using neuron-level concepts to enable practitioners to have a high-level view of the model. Specifically, (i) to the best of our knowledge, this is the first work which shows that GNN neurons act as concept detectors and have strong alignment with concepts formulated as logical compositions of node degree and neighbourhood properties; (ii) we quantitatively assess the importance of detected concepts, and identify a trade-off between training duration and neuron-level interpretability; (iii) we demonstrate that our global explainability approach has advantages over the current state-of-the-art -- we can disentangle the explanation into individual interpretable concepts backed by logical descriptions, which reduces potential for bias and improves user-friendliness.