mlsanityData to 7 Oct 2026

Evaluating Readability and Faithfulness of Concept-based Explanations

arXiv 2404.18533 ↗First posted 29 Apr 2024cs.AI cs.HCcited by 10

Not on the map. Why?

Abstract (arXiv)

With the growing popularity of general-purpose Large Language Models (LLMs), comes a need for more global explanations of model behaviors. Concept-based explanations arise as a promising avenue for explaining high-level patterns learned by LLMs. Yet their evaluation poses unique challenges, especially due to their non-local nature and high dimensional representation in a model's hidden space. Current methods approach concepts from different perspectives, lacking a unified formalization. This makes evaluating the core measures of concepts, namely faithfulness or readability, challenging. To bridge the gap, we introduce a formal definition of concepts generalizing to diverse concept-based explanations' settings. Based on this, we quantify the faithfulness of a concept explanation via perturbation. We ensure adequate perturbation in the high-dimensional space for different concepts via an optimization problem. Readability is approximated via an automatic and deterministic measure, quantifying the coherence of patterns that maximally activate a concept while aligning with human understanding. Finally, based on measurement theory, we apply a meta-evaluation method for evaluating these measures, generalizable to other types of explanations or tasks as well. Extensive experimental analysis has been conducted to inform the selection of explanation evaluation measures.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 47 of its 79 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

47 of 79 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  10. 2018Sanity Checks for Saliency Mapscited by 1,149
  11. 2020Concept Bottleneck Modelscited by 1,022
  12. 2022Toy Models of Superpositioncited by 928
35 more
  1. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  2. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  3. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  4. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  5. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  6. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  7. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  8. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  9. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  10. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  11. 2020Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectorscited by 119
  12. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  13. 2019CXPlain: Causal Explanations for Model Interpretation under Uncertaintycited by 98
  14. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  15. 2019An Evaluation of the Human-Interpretability of Explanationcited by 71
  16. 2023Explaining black box text modules in natural language with language modelscited by 61
  17. 2019Theory and Evaluation Metrics for Learning Disentangled Representationscited by 57
  18. 2022Discovering Latent Concepts Learned in BERTcited by 56
  19. 2023Learning Bottleneck Concepts in Image Classificationcited by 49
  20. 2020Intrinsic Probing through Dimension Selectioncited by 45
  21. 2021A Framework for Learning Ante-hoc Explainable Models via Conceptscited by 45
  22. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  23. 2022A Comparative Study of Faithfulness Metrics for Model Interpretability Methodscited by 38
  24. 2021DISSECT: Disentangled Simultaneous Explanations via Concept Traversalscited by 37
  25. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  26. 2023Explain Any Concept: Segment Anything Meets Concept-Based Explanationcited by 37
  27. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  28. 2022Understanding and Enhancing Robustness of Concept-based Modelscited by 22
  29. 2019Discovery of Natural Language Concepts in Individual Units of CNNscited by 21
  30. 2022A Unified Understanding of Deep NLP Models for Text Classificationcited by 18
  31. 2022Self-explaining deep models with logic rule reasoningcited by 13
  32. 2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networkscited by 13
  33. 2023Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainabilitycited by 11
  34. 2023Are Your Explanations Reliable? Investigating the Stability of LIME in Explaining Text Classifiers by Marrying XAI and Adversarial Attackcited by 10
  35. 2020Abstracting Deep Neural Networks into Concept Graphs for Concept Level Interpretabilitycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Mechanistic understanding and validation of large AI models with SemanticLenscited by 29
  2. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  3. 2025SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Interventioncited by 4
  4. 2025Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoderscited by 3
  5. 2025Interpreting Language Models Through Concept Descriptions: A Surveycited by 3
  6. 2026Enhancing Safety of Large Language Models via Embedding Space Separationcited by 2
  7. 2026Do LLMs Know Their Vulnerable Scenarios?cited by 0
  8. 2026ICON Decomposition: Auditing deep neural networks for shortcuts by decomposing layer-wise representations using conceptscited by 0
  9. 2026Analyzing Traditional and Neural Approaches to Multilingual Readability Assessmentcited by 0
  10. 2026A Unifying Framework of Concept-based Explainable AI with Completeness Guaranteescited by 0