Data to 7 Oct 2026

Analyzing Individual Neurons in Pre-trained Language Models

arXiv 2010.02695 ↗First posted 6 Oct 2020cs.CLcited by 97

Not on the map. Why?

Abstract (arXiv)

While a lot of analysis has been carried to demonstrate linguistic knowledge captured by the representations learned within deep NLP models, very little attention has been paid towards individual neurons.We carry outa neuron-level analysis using core linguistic tasks of predicting morphology, syntax and semantics, on pre-trained language models, with questions like: i) do individual neurons in pre-trained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how distributed or focused is the information? and iv) how do various architectures differ in learning these properties? We found small subsets of neurons to predict linguistic tasks, with lower level tasks (such as morphology) localized in fewer neurons, compared to higher level task of predicting syntax. Our study also reveals interesting cross architectural comparisons. For example, we found neurons in XLNet to be more localized and disjoint when predicting properties compared to BERT and others, where they are more distributed and coupled.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 19; 2023: 16; 2024: 18; 2025: 19; 2026: 6.

97 papers.

Sort:

By area:

2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
2022Analyzing Transformers in Embedding Spacecited by 131
2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
2021An Interpretability Illusion for BERTcited by 94
2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65

38 more

2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61
2022Discovering Latent Concepts Learned in BERTcited by 56
2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
2022Interpreting Neural Networks through the Polytope Lenscited by 33
2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
2021Improving Transformers with Probabilistic Attention Keyscited by 28
2023Dissociating language and thought in large language modelscited by 27
2023Neuron to Graph: Interpreting Language Model Neurons at Scalecited by 26
2024Large Language Models Relearn Removed Conceptscited by 26
2024Mechanistic?cited by 25
2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
2025RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patternscited by 24
2021A Bayesian Framework for Information-Theoretic Probingcited by 22
2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
2021Of Non-Linearity and Commutativity in BERTcited by 16
2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
2023Towards a fuller understanding of neurons with Clustered Compositional Explanationscited by 15
2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
2022Probing for Constituency Structure in Neural Language Modelscited by 13
2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networkscited by 13
2025Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captionscited by 13
2022Measures of Information Reflect Memorization Patternscited by 12
2024Identifying Query-Relevant Neurons in Large Language Models for Long-Form Textscited by 12
2021Probing as Quantifying Inductive Biascited by 11
2023NeuroX Library for Neuron Analysis of Deep NLP Modelscited by 11
2025Language-specific Neurons Do Not Facilitate Cross-Lingual Transfercited by 11
2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
2023SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classificationcited by 10
2025Towards Understanding How Knowledge Evolves in Large Vision-Language Modelscited by 10
2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9

50 most cited of 97 All 97 (JSON)