mlsanityData to 14 Oct 2026

From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks

arXiv 2310.11884 ↗First posted 18 Oct 2023cs.AI cs.CL cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we review recent approaches for explaining concepts in neural networks. Concepts can act as a natural link between learning and reasoning: once the concepts are identified that a neural learning system uses, one can integrate those concepts with a reasoning system for inference or use a reasoning system to act upon them to improve or enhance the learning system. On the other hand, knowledge can not only be extracted from neural networks but concept knowledge can also be inserted into neural network architectures. Since integrating learning and reasoning is at the core of neuro-symbolic AI, the insights gained from this survey can serve as an important step towards realizing neuro-symbolic AI based on explainable concepts.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 38 of its 62 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

38 of 62 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  7. 2020Concept Bottleneck Modelscited by 1,032
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  12. 2022Post-hoc Concept Bottleneck Modelscited by 293
26 more
  1. 2023Label-Free Concept Bottleneck Modelscited by 285
  2. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  3. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  4. 2020Compositional Explanations of Neuronscited by 172
  5. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  6. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
  7. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  8. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  9. 2018Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networkscited by 129
  10. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  11. 2020Explainable Deep Learning: A Field Guide for the Uninitiatedcited by 127
  12. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
  13. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  14. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  15. 2023Interpretable Neural-Symbolic Concept Reasoningcited by 57
  16. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  17. 2021Logic Explained Networkscited by 45
  18. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  19. 2020Explainable Goal-Driven Agents and Robots -- A Comprehensive Reviewcited by 24
  20. 2019Discovery of Natural Language Concepts in Individual Units of CNNscited by 21
  21. 2022Concept Embedding Analysis: A Reviewcited by 20
  22. 2019On the Units of GANs (Extended Abstract)cited by 19
  23. 2022Naturalistic Causal Probing for Morpho-Syntaxcited by 19
  24. 2022Improving Generalizability in Implicitly Abusive Language Detection with Concept Activation Vectorscited by 14
  25. 2023The Roles of Symbols in Neural-based AI: They are Not What You Think!cited by 4
  26. 2023Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  2. 2024Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?cited by 18
  3. 2024Evaluating Readability and Faithfulness of Concept-based Explanationscited by 10
  4. 2023Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spacescited by 7
  5. 2024Incorporating Expert Rules into Neural Networks in the Framework of Concept-Based Learningcited by 3
  6. 2025Interpreting Language Models Through Concept Descriptions: A Surveycited by 3
  7. 2024GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifierscited by 2
  8. 2026Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoderscited by 2
  9. 2024FI-CBL: A Probabilistic Method for Concept-Based Learning with Expert Rulescited by 1
  10. 2025On Background Bias of Post-Hoc Concept Embeddings in Computer Vision DNNscited by 1
  11. 2025Survival Concept-Based Learning Modelscited by 0
  12. 2025Neural Concept Verifier: Scaling Prover-Verifier Games via Concept Encodingscited by 0
1 more
  1. 2025On the Performance of Concept Probing: The Influence of the Data (Extended Version)cited by 0