mlsanityData to 7 Oct 2026

Rigorously Assessing Natural Language Explanations of Neurons

arXiv 2309.10312 ↗First posted 19 Sep 2023cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging. To help address this, we develop two modes of evaluation for natural language explanations that claim individual neurons represent a concept in a text input. In the observational mode, we evaluate claims that a neuron a activates on all and only input strings that refer to a concept picked out by the proposed explanation E. In the intervention mode, we construe E as a claim that the neuron a is a causal mediator of the concept denoted by E. We apply our framework to the GPT-4-generated explanations of GPT-2 XL neurons of Bills et al. (2023) and show that even the most confident explanations have high error rates and little to no causal efficacy. We close the paper by critically assessing whether natural language is a good choice for explanations and whether neurons are the best level of analysis.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

32 of 45 references.

References · 32

  1. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2022Mass-Editing Memory in a Transformercited by 976
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2022Toy Models of Superpositioncited by 928
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
20 more
  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  2. 2021Causal Abstractions of Neural Networkscited by 401
  3. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  4. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  5. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  6. 2020Compositional Explanations of Neuronscited by 172
  7. 2016Generating Visual Explanationscited by 156
  8. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  9. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  10. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  11. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  12. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  13. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  14. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61
  15. 2023Explaining black box text modules in natural language with language modelscited by 61
  16. 2022Disentangling visual and written concepts in CLIPcited by 57
  17. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  18. 2022What do tokens know about their characters and how do they know it?cited by 36
  19. 2022Explaining Chest X-ray Pathologies in Natural Languagecited by 27
  20. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 19; 2025: 12; 2026: 12.

44 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  3. 2024How to use and interpret activation patchingcited by 191
  4. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  5. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  6. 2024Universal Neurons in GPT2 Language Modelscited by 100
  7. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  8. 2024A Multimodal Automated Interpretability Agentcited by 62
  9. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  10. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  11. 2024Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoderscited by 25
  12. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
32 more
  1. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 21
  2. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 20
  3. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  4. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  5. 2024A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Argumentscited by 15
  6. 2024Mitigating Memorization In Language Modelscited by 11
  7. 2025Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviorscited by 11
  8. 2024Natural Language Processing RELIES on Linguisticscited by 9
  9. 2025Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Frameworkcited by 9
  10. 2025HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworkscited by 8
  11. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 8
  12. 2025Discovering Influential Neuron Path in Vision Transformerscited by 7
  13. 2024Anthropocentric bias in language model evaluationcited by 4
  14. 2025Evaluating Neuron Explanations: A Unified Framework with Sanity Checkscited by 4
  15. 2025Measuring Sparse Autoencoder Feature Sensitivitycited by 4
  16. 2025Faithful and Stable Neuron Explanations for Trustworthy Mechanistic Interpretabilitycited by 3
  17. 2026Endogenous Resistance to Activation Steering in Language Modelscited by 3
  18. 2025Semantic Regexes: Auto-Interpreting LLM Features with a Structured Languagecited by 2
  19. 2025Automated Detection of Visual Attribute Reliance with a Self-Reflective Agentcited by 2
  20. 2026Pitfalls in Evaluating Interpretability Agentscited by 2
  21. 2026Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many Featurescited by 2
  22. 2026Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanationscited by 2
  23. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  24. 2026Disentangling MLP Neuron Weights in Vocabulary Spacecited by 1
  25. 2026Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attributioncited by 1
  26. 2026How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordingscited by 1
  27. 2024Deep Model Merging: The Sister of Neural Network Interpretability -- A Surveycited by 0
  28. 2025VERBA: Verbalizing Model Differences Using Large Language Modelscited by 0
  29. 2026Steering grids for sparse-autoencoder features: when a top-context label names an activation regime rather than a causal axiscited by 0
  30. 2026Position: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claimscited by 0
  31. 2026Measuring Semantic Abstractness of SAE Features via Nonlocalitycited by 0
  32. 2026Circuit-Diff: Factual Edit-based Intervention Method for Localizing Knowledge in Attribution Graphscited by 0