# Evaluating Readability and Faithfulness of Concept-based Explanations

arXiv: https://arxiv.org/abs/2404.18533 · HTML: https://mlsanity.com/paper/2404.18533/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 79 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors](https://mlsanity.com/paper/2006.15417/) (2020)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)
- [CXPlain: Causal Explanations for Model Interpretation under Uncertainty](https://mlsanity.com/paper/1910.12336/) (2019)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1902.00006/) (2019)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Theory and Evaluation Metrics for Learning Disentangled Representations](https://mlsanity.com/paper/1908.09961/) (2019)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)

## Built on by (most cited first, 10 of 10)

- [Mechanistic understanding and validation of large AI models with SemanticLens](https://mlsanity.com/paper/2501.05398/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Intervention](https://mlsanity.com/paper/2502.15594/) (2025)
- [Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders](https://mlsanity.com/paper/2506.23951/) (2025)
- [Interpreting Language Models Through Concept Descriptions: A Survey](https://mlsanity.com/paper/2510.01048/) (2025)
- [Enhancing Safety of Large Language Models via Embedding Space Separation](https://mlsanity.com/paper/2603.20206/) (2026)
- [Do LLMs Know Their Vulnerable Scenarios?](https://mlsanity.com/paper/2607.23496/) (2026)
- [ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on](https://mlsanity.com/paper/2608.26083/) (2026)
- [Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment](https://mlsanity.com/paper/2609.10792/) (2026)
- [A Unifying Framework of Concept-based Explainable AI with Completeness Guarantees](https://mlsanity.com/paper/2609.34750/) (2026)

## Abstract (arXiv)

With the growing popularity of general-purpose Large Language Models (LLMs), comes a need for more global explanations of model behaviors. Concept-based explanations arise as a promising avenue for explaining high-level patterns learned by LLMs. Yet their evaluation poses unique challenges, especially due to their non-local nature and high dimensional representation in a model's hidden space. Current methods approach concepts from different perspectives, lacking a unified formalization. This makes evaluating the core measures of concepts, namely faithfulness or readability, challenging. To bridge the gap, we introduce a formal definition of concepts generalizing to diverse concept-based explanations' settings. Based on this, we quantify the faithfulness of a concept explanation via perturbation. We ensure adequate perturbation in the high-dimensional space for different concepts via an optimization problem. Readability is approximated via an automatic and deterministic measure, quantifying the coherence of patterns that maximally activate a concept while aligning with human understanding. Finally, based on measurement theory, we apply a meta-evaluation method for evaluating these measures, generalizable to other types of explanations or tasks as well. Extensive experimental analysis has been conducted to inform the selection of explanation evaluation measures.