mlsanityData to 7 Oct 2026

On the Pitfalls of Analyzing Individual Neurons in Language Models

arXiv 2110.07483 ↗First posted 14 Oct 2021cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is encoded. Among these, the common approach is to use an external probe to rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it. We show two pitfalls in this methodology: 1. It confounds distinct factors: probe quality and ranking quality. We separate them and draw conclusions on each. 2. It focuses on encoded information, rather than information that is used by the model. We show that these are not the same. We compare two recent ranking methods and a simple one we introduce, and evaluate them with regard to both of these aspects.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

18 of 27 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2019Designing and Interpreting Probes with Control Taskscited by 608
  5. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  6. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  7. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  8. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  9. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  10. 2018On the importance of single directions for generalizationcited by 99
  11. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  12. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
6 more
  1. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  2. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  3. 2020Intrinsic Probing through Dimension Selectioncited by 45
  4. 2018UniMorph 2.0: Universal Morphologycited by 40
  5. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  6. 2018Marrying Universal Dependencies and Universal Morphologycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 11; 2024: 16; 2025: 14; 2026: 8.

61 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  3. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  5. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  6. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  7. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  8. 2024Universal Neurons in GPT2 Language Modelscited by 100
  9. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  10. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  11. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  12. 2024Confidence Regulation Neurons in Language Modelscited by 57
38 more
  1. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  2. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  3. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  4. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  5. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  6. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
  7. 2024Large Language Models Relearn Removed Conceptscited by 26
  8. 2024Mechanistic?cited by 25
  9. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  10. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  11. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  12. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  13. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  14. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  15. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 15
  16. 2023Towards a fuller understanding of neurons with Clustered Compositional Explanationscited by 15
  17. 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
  18. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  19. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  20. 2022Measures of Information Reflect Memorization Patternscited by 12
  21. 2021Probing as Quantifying Inductive Biascited by 11
  22. 2023NeuroX Library for Neuron Analysis of Deep NLP Modelscited by 11
  23. 2022Analyzing Gender Representation in Multilingual Modelscited by 10
  24. 2023Evaluating Neuron Interpretation Methods of NLP Modelscited by 8
  25. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 8
  26. 2024Crafting Large Language Models for Enhanced Interpretabilitycited by 7
  27. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  28. 2026Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Modelscited by 6
  29. 2022A Latent-Variable Model for Intrinsic Probingcited by 5
  30. 2025Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attributioncited by 5
  31. 2022IDANI: Inference-time Domain Adaptation via Neuron-level Interventionscited by 4
  32. 2025On Relation-Specific Neurons in Large Language Modelscited by 4
  33. 2025Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognitioncited by 4
  34. 2024Towards detecting unanticipated bias in Large Language Modelscited by 3
  35. 2025Evaluating Sparse Autoencoders for Monosemantic Representationcited by 3
  36. 2025Continual Personalization for Diffusion Modelscited by 3
  37. 2023DeepDecipher: Accessing and Investigating Neuron Activation in Large Language Modelscited by 2
  38. 2025Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddingscited by 2

50 most cited of 61 All 61 (JSON)