mlsanityData to 7 Oct 2026

Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience

arXiv 2406.01352 ↗First posted 3 Jun 2024cs.AI cs.LG q-bio.NCcited by 11

Not on the map. Why?

Abstract (arXiv)

Inner Interpretability is a promising emerging field tasked with uncovering the inner mechanisms of AI systems, though how to develop these mechanistic theories is still much debated. Moreover, recent critiques raise issues that question its usefulness to advance the broader goals of AI. However, it has been overlooked that these issues resemble those that have been grappled with in another field: Cognitive Neuroscience. Here we draw the relevant connections and highlight lessons that can be transferred productively between fields. Based on these, we propose a general conceptual framework and give concrete methodological strategies for building mechanistic explanations in AI inner interpretability research. With this conceptual framework, Inner Interpretability can fend off critiques and position itself on a productive path to explain AI systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 48 of its 98 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

48 of 98 references.

References · 48

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2019How Can We Know What Language Models Know?cited by 953
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2022Toy Models of Superpositioncited by 928
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
36 more
  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2023Function Vectors in Large Language Modelscited by 292
  5. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  6. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  7. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  8. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  9. 2020Compositional Explanations of Neuronscited by 172
  10. 2020The Hardware Lotterycited by 160
  11. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  12. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  13. 2023Getting aligned on representational alignmentcited by 142
  14. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  15. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  16. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  17. 2017Causal Consistency of Structural Equation Modelscited by 114
  18. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  19. 2018Abstracting Causal Modelscited by 104
  20. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  21. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  22. 2021An Interpretability Illusion for BERTcited by 94
  23. 2019Augmenting Self-attention with Persistent Memorycited by 92
  24. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  25. 2023Into the LAIONs Den: Investigating Hate in Multimodal Datasetscited by 54
  26. 2023Characterizing Mechanisms for Factual Recall in Language Modelscited by 47
  27. 2023Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learningcited by 42
  28. 2021Why AI is Harder Than We Thinkcited by 40
  29. 2023Neither hype nor gloom do DNNs justicecited by 40
  30. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 39
  31. 2023Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuitscited by 26
  32. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  33. 2023Uncovering Intermediate Variables in Transformers using Circuit Probingcited by 12
  34. 2020Levels of Analysis for Machine Learningcited by 11
  35. 2019Deep Learning for Cognitive Neurosciencecited by 7
  36. 2023Instilling Inductive Biases with Subnetworkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 2.

11 papers.

  1. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  2. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  3. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  4. 2024Understanding Reasoning in Chain-of-Thought from the Hopfieldian Viewcited by 10
  5. 2025Mechanistic Interpretability Needs Philosophycited by 6
  6. 2024Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neurosciencecited by 5
  7. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  8. 2025Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Modelscited by 4
  9. 2025Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretabilitycited by 1
  10. 2024Bi-ICE: An Inner Interpretable Framework for Image Classification via Bi-directional Interactions between Concept and Input Embeddingscited by 0
  11. 2026(How) Do MLLMs Report Bistable Images Like Humans?cited by 0