# When concept-based XAI is imprecise: Do people distinguish between generalisations and misrepresentations?

arXiv: https://arxiv.org/abs/2506.17936 · HTML: https://mlsanity.com/paper/2506.17936/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 22 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Object Detectors Emerge in Deep Scene CNNs](https://mlsanity.com/paper/1412.6856/) (2014)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors](https://mlsanity.com/paper/2006.15417/) (2020)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [Learning Bottleneck Concepts in Image Classification](https://mlsanity.com/paper/2304.10131/) (2023)
- [Mechanistic understanding and validation of large AI models with SemanticLens](https://mlsanity.com/paper/2501.05398/) (2025)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [OSDaR23: Open Sensor Data for Rail 2023](https://mlsanity.com/paper/2305.03001/) (2023)
- [Describing Common Human Visual Actions in Images](https://mlsanity.com/paper/1506.02203/) (2015)
- [Embedding Deep Networks into Visual Explanations](https://mlsanity.com/paper/1709.05360/) (2017)
- [Visual representations in the human brain are aligned with large language models](https://mlsanity.com/paper/2209.11737/) (2022)
- [Cross-Lingual and Cross-Cultural Variation in Image Descriptions](https://mlsanity.com/paper/2409.16646/) (2024)
- [RailGoerl24: G"orlitz Rail Test Center CV Dataset 2024](https://mlsanity.com/paper/2504.00204/) (2025)
- [Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categories](https://mlsanity.com/paper/2308.10562/) (2023)
- [Beyond object identification: How train drivers evaluate the risk of collision](https://mlsanity.com/paper/2411.10475/) (2024)
- [How humans evaluate AI systems for person detection in automatic train operation: Not all misses are alike](https://mlsanity.com/paper/2504.02664/) (2025)

## Abstract (arXiv)

Concept-based explainable artificial intelligence (C-XAI) can let people see which representations an AI model has learned. This is particularly important when high-level semantic information (e.g., actions and relations) is used to make decisions about abstract categories (e.g., danger). In such tasks, AI models need to generalise beyond situation-specific details, and this ability can be reflected in C-XAI outputs that randomise over irrelevant features. However, it is unclear whether people appreciate such generalisation and can distinguish it from other, less desirable forms of imprecision in C-XAI outputs. Therefore, the present study investigated how the generality and relevance of C-XAI outputs affect people's evaluation of AI. In an experimental railway safety evaluation scenario, participants rated the performance of a simulated AI that classified traffic scenes involving people as dangerous or not. These classification decisions were explained via concepts in the form of similar image snippets. The latter differed in their match with the classified image, either regarding a highly relevant feature (i.e., people's relation to tracks) or a less relevant feature (i.e., people's action). Contrary to the hypotheses, concepts that generalised over less relevant features were rated lower than concepts that matched the classified image precisely. Moreover, their ratings were no better than those for systematic misrepresentations of the less relevant feature. Conversely, participants were highly sensitive to imprecisions in relevant features. These findings cast doubts on the assumption that people can easily infer from C-XAI outputs whether AI models have gained a deeper understanding of complex situations.