# Human-Centered Concept Explanations for Neural Networks

arXiv: https://arxiv.org/abs/2202.12451 · HTML: https://mlsanity.com/paper/2202.12451/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 88 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)

## Built on by (most cited first, 23 of 23)

- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Xplique: A Deep Learning Explainability Toolbox](https://mlsanity.com/paper/2206.04394/) (2022)
- [Unpacking Large Language Models with Conceptual Consistency](https://mlsanity.com/paper/2209.15093/) (2022)
- [Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement](https://mlsanity.com/paper/2311.15303/) (2023)
- [Towards Modeling Uncertainties of Self-explaining Neural Networks via Conformal Prediction](https://mlsanity.com/paper/2401.01549/) (2024)
- [Object Centric Concept Bottlenecks](https://mlsanity.com/paper/2505.24492/) (2025)
- [A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities](https://mlsanity.com/paper/2210.08906/) (2022)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [Neural Activation Patterns (NAPs): Visual Explainability of Learned Concepts](https://mlsanity.com/paper/2206.10611/) (2022)
- [Towards Procedural Fairness: Uncovering Biases in How a Toxic Language Classifier Uses Sentiment Information](https://mlsanity.com/paper/2210.10689/) (2022)
- [Concept-based Analysis of Neural Networks via Vision-Language Models](https://mlsanity.com/paper/2403.19837/) (2024)
- [Provably Better Explanations with Optimized Aggregation of Feature Attributions](https://mlsanity.com/paper/2406.05090/) (2024)
- [Validating Mechanistic Interpretations: An Axiomatic Approach](https://mlsanity.com/paper/2407.13594/) (2024)
- [Learning Unsupervised Hierarchies of Audio Concepts](https://mlsanity.com/paper/2207.11231/) (2022)
- [On the Role of Emergent Communication for Social Learning in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.14276/) (2023)
- [Concept-Based Explanations to Test for False Causal Relationships Learned by Abusive Language Classifiers](https://mlsanity.com/paper/2307.01900/) (2023)
- [Debugging and Runtime Analysis of Neural Networks with VLMs (A Case Study)](https://mlsanity.com/paper/2503.17416/) (2025)
- [A Model for Intelligible Interaction Between Agents That Predict and Explain](https://mlsanity.com/paper/2301.01819/) (2023)
- [Explaining Deep Neural Networks for Bearing Fault Detection with Vibration Concepts](https://mlsanity.com/paper/2310.11450/) (2023)
- [Analyzing Latent Concepts in Code Language Models](https://mlsanity.com/paper/2510.00476/) (2025)
- [Estimation of Concept Explanations Should be Uncertainty Aware](https://mlsanity.com/paper/2312.08063/) (2023)
- [HEX: Human-in-the-loop Explainability via Deep Reinforcement Learning](https://mlsanity.com/paper/2206.01343/) (2022)
- [Out-of-Distribution Detection via Channelwise Feature Aggregation in Neural Network-Based Receivers](https://mlsanity.com/paper/2505.15570/) (2025)

## Abstract (arXiv)

Understanding complex machine learning models such as deep neural networks with explanations is crucial in various applications. Many explanations stem from the model perspective, and may not necessarily effectively communicate why the model is making its predictions at the right level of abstraction. For example, providing importance weights to individual pixels in an image can only express which parts of that particular image are important to the model, but humans may prefer an explanation which explains the prediction by concept-based thinking. In this work, we review the emerging area of concept based explanations. We start by introducing concept explanations including the class of Concept Activation Vectors (CAV) which characterize concepts using vectors in appropriate spaces of neural activations, and discuss different properties of useful concepts, and approaches to measure the usefulness of concept vectors. We then discuss approaches to automatically extract concepts, and approaches to address some of their caveats. Finally, we discuss some case studies that showcase the utility of such concept-based explanations in synthetic settings and real world applications.