mlsanityData to 7 Oct 2026

Neuron-level Interpretation of Deep NLP Models: A Survey

arXiv 2108.13138 ↗First posted 30 Aug 2021cs.CLcited by 81

Not on the map. Why?

Abstract (arXiv)

The proliferation of deep neural networks in various domains has seen an increased need for interpretability of these models. Preliminary work done along this line and papers that surveyed such, are focused on high-level representation analysis. However, a recent branch of work has concentrated on interpretability at a more granular level of analyzing neurons within these models. In this paper, we survey the work done on neuron analysis including: i) methods to discover and understand neurons in a network, ii) evaluation methods, iii) major findings including cross architectural comparisons that neuron analysis has unraveled, iv) applications of neuron probing such as: controlling the model, domain adaptation etc., and v) a discussion on open issues and future research directions.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 57 of its 88 references that are in the map. First posted on arXiv 30 Aug 2021.

Builds on

57 of 88 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2016The Mythos of Model Interpretabilitycited by 716
38 more
  1. 2019Designing and Interpreting Probes with Control Taskscited by 608
  2. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  3. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  4. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  5. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  6. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  7. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  8. 2016Understanding Neural Networks through Representation Erasurecited by 223
  9. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  10. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  11. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  12. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  13. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  14. 2020Compositional Explanations of Neuronscited by 172
  15. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  16. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  17. 2015Visualizing and Understanding Recurrent Networkscited by 158
  18. 2019The Shapley Taylor Interaction Indexcited by 143
  19. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  20. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  21. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  22. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  23. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  24. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  25. 2018How Important Is a Neuron?cited by 73
  26. 2018Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMscited by 67
  27. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  28. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  29. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  30. 2020Intrinsic Probing through Dimension Selectioncited by 45
  31. 2015Sparse Overcomplete Word Vector Representationscited by 43
  32. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  33. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  34. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  35. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  36. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  37. 2020Finding Experts in Transformer Modelscited by 28
  38. 2019Discovery of Natural Language Concepts in Individual Units of CNNscited by 21

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 19; 2024: 26; 2025: 19; 2026: 6.

81 papers.

Sort:

By area:

  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  5. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  6. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  7. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  8. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  9. 2022Discovering Latent Concepts Learned in BERTcited by 56
  10. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  11. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  12. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
38 more
  1. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  2. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  3. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  4. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  5. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  6. 2024Mechanistic?cited by 25
  7. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  8. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  9. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 23
  10. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  11. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  12. 2024Linear Explanations for Individual Neuronscited by 19
  13. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  14. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  15. 2024CascadedGaze: Efficiency in Global Context Extraction for Image Restorationcited by 16
  16. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 15
  17. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  18. 2023Towards a fuller understanding of neurons with Clustered Compositional Explanationscited by 15
  19. 2024LLM Roleplay: Simulating Human-Chatbot Interactioncited by 15
  20. 2022Probing for Constituency Structure in Neural Language Modelscited by 13
  21. 2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networkscited by 13
  22. 2025LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encodercited by 13
  23. 2023The Importance of Prompt Tuning for Automated Neuron Explanationscited by 12
  24. 2023Cross-Modal Conceptualization in Bottleneck Modelscited by 12
  25. 2023NeuroX Library for Neuron Analysis of Deep NLP Modelscited by 11
  26. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  27. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  28. 2022Impact of Adversarial Training on Robustness and Generalizability of Language Modelscited by 9
  29. 2023Scaling up Discovery of Latent Concepts in Deep NLP Modelscited by 9
  30. 2024Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methodscited by 9
  31. 2023Evaluating Neuron Interpretation Methods of NLP Modelscited by 8
  32. 2022ConceptX: A Framework for Latent Concept Analysiscited by 6
  33. 2025Concept-Level Explainability for Auditing & Steering LLM Responsescited by 6
  34. 2023Multilingual Nonce Dependency Treebanks: Understanding how Language Models represent and process syntactic structurecited by 5
  35. 2025Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attributioncited by 5
  36. 2022IDANI: Inference-time Domain Adaptation via Neuron-level Interventionscited by 4
  37. 2024Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Modelscited by 4
  38. 2025On Relation-Specific Neurons in Large Language Modelscited by 4

50 most cited of 81 All 81 (JSON)