mlsanityData to 7 Oct 2026

Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

arXiv 1711.11279 ↗First posted 30 Nov 2017stat.MLcited by 1,309

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of human-friendly concepts. The key idea is to view the high-dimensional internal state of a neural net as an aid, not an obstacle. We show how to use CAVs as part of a technique, Testing with CAVs (TCAV), that uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result--for example, how sensitive a prediction of "zebra" is to the presence of stripes. Using the domain of image classification as a testing ground, we describe how CAVs may be used to explore hypotheses and generate insights for a standard image classification network as well as a medical application.

Key facts. Cited by 1,309 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 149; 2022: 191; 2023: 236; 2024: 242; 2025: 240; 2026: 250.

1,309 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  4. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  5. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  6. 2023Explainability for Large Language Models: A Surveycited by 331
  7. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  8. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  9. 2022Post-hoc Concept Bottleneck Modelscited by 289
  10. 2023Label-Free Concept Bottleneck Modelscited by 282
  11. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
38 more
  1. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  2. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  3. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  4. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  5. 2021Explainable artificial intelligence (XAI) in deep learning-based medical image analysiscited by 147
  6. 2023Getting aligned on representational alignmentcited by 142
  7. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  8. 2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
  9. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  10. 2021Promises and Pitfalls of Black-Box Concept Learning Modelscited by 128
  11. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  12. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  13. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  14. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  15. 2021Explaining in Style: Training a GAN to explain a classifier in StyleSpacecited by 122
  16. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  17. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  18. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  19. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  20. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  21. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  22. 2021Guided Integrated Gradients: An Adaptive Path Method for Removing Noisecited by 102
  23. 2023Probabilistic Concept Bottleneck Modelscited by 101
  24. 2022Linear Adversarial Concept Erasurecited by 98
  25. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  26. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 97
  27. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  28. 2021An Interpretability Illusion for BERTcited by 94
  29. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  30. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
  31. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  32. 2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 89
  33. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  34. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  35. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  36. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  37. 2021Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methodscited by 83
  38. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83

50 most cited of 1,309 All 1,309 (JSON)