# Explaining Explainability: Recommendations for Effective Use of Concept Activation Vectors

arXiv: https://arxiv.org/abs/2404.03713 · HTML: https://mlsanity.com/paper/2404.03713/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2024.

## Built on by (most cited first, 13 of 13)

- [Concept Bottleneck Models Without Predefined Concepts](https://mlsanity.com/paper/2407.03921/) (2024)
- [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025)
- [Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs](https://mlsanity.com/paper/2505.15524/) (2025)
- [The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts](https://mlsanity.com/paper/2604.11962/) (2026)
- [In-hoc Concept Representations to Regularise Deep Learning in Medical Imaging](https://mlsanity.com/paper/2508.13880/) (2025)
- [FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI](https://mlsanity.com/paper/2511.15481/) (2025)
- [Explainable AI Methods for Multi-Omics Analysis: A Survey](https://mlsanity.com/paper/2410.11910/) (2024)
- [Concept Boundary Vectors](https://mlsanity.com/paper/2412.15698/) (2024)
- [Probing the Probes: Methods and Metrics for Concept Alignment](https://mlsanity.com/paper/2511.04312/) (2025)
- [E-TCAV: Formalizing Penultimate Proxies for Efficient Concept Based Interpretability](https://mlsanity.com/paper/2605.10261/) (2026)
- [α-TCAV: A Unified Framework for Testing with Concept Activation Vectors](https://mlsanity.com/paper/2605.15688/) (2026)
- [On the Faithfulness of Post-Hoc Concept Bottleneck Models](https://mlsanity.com/paper/2606.30498/) (2026)
- [ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on](https://mlsanity.com/paper/2608.26083/) (2026)

## Abstract (arXiv)

Concept-based explanations translate the internal representations of deep learning models into a language that humans are familiar with: concepts. One popular method for finding concepts is Concept Activation Vectors (CAVs), which are learnt using a probe dataset of concept exemplars. In this work, we investigate three properties of CAVs: (1) inconsistency across layers, (2) entanglement with other concepts, and (3) spatial dependency. Each property provides both challenges and opportunities in interpreting models. We introduce tools designed to detect the presence of these properties, provide insight into how each property can lead to misleading explanations, and provide recommendations to mitigate their impact. To demonstrate practical applications, we apply our recommendations to a melanoma classification task, showing how entanglement can lead to uninterpretable results and that the choice of negative probe set can have a substantial impact on the meaning of a CAV. Further, we show that understanding these properties can be used to our advantage. For example, we introduce spatially dependent CAVs to test if a model is translation invariant with respect to a specific concept and class. Our experiments are performed on natural images (ImageNet), skin lesions (ISIC 2019), and a new synthetic dataset, Elements. Elements is designed to capture a known ground truth relationship between concepts and classes. We release this dataset to facilitate further research in understanding and evaluating interpretability methods.