mlsanityData to 7 Oct 2026

Universal Neurons in GPT2 Language Models

arXiv 2401.12181 ↗First posted 22 Jan 2024cs.LG cs.AI cs.CLcited by 100

Not on the map. Why?

Abstract (arXiv)

A basic question within the emerging field of mechanistic interpretability is the degree to which neural networks learn the same underlying mechanisms. In other words, are neural mechanisms universal across different models? In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable. In particular, we compute pairwise correlations of neuron activations over 100 million tokens for every neuron pair across five different seeds and find that 1-5% of neurons are universal, that is, pairs of neurons which consistently activate on the same inputs. We then study these universal neurons in detail, finding that they usually have clear interpretations and taxonomize them into a small number of neuron families. We conclude by studying patterns in neuron weights to establish several universal functional roles of neurons in simple circuits: deactivating attention heads, changing the entropy of the next token distribution, and predicting the next token to (not) be within a particular set.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 78 of its 103 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

78 of 103 references.

References · 78

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
38 more
  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2022Toy Models of Superpositioncited by 928
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  9. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  13. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  14. 2023Language Models Represent Space and Timecited by 306
  15. 2023In-Context Learning Creates Task Vectorscited by 302
  16. 2023Function Vectors in Large Language Modelscited by 292
  17. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  18. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  19. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  20. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  21. 2023An Overview of Catastrophic AI Riskscited by 214
  22. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  23. 2020Compositional Explanations of Neuronscited by 172
  24. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  25. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  26. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  27. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  28. 2022Analyzing Transformers in Embedding Spacecited by 131
  29. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  30. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  31. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  32. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  33. 2021Generalized Shape Metrics on Neural Representationscited by 109
  34. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  35. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  36. 2023How do Language Models Bind Entities in Context?cited by 102
  37. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  38. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 40; 2026: 29.

100 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  4. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  5. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  6. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  7. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  8. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  9. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  10. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  11. 2024Confidence Regulation Neurons in Language Modelscited by 57
  12. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
38 more
  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  3. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  4. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  5. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  6. 2025Training Language Models to Explain Their Own Computationscited by 27
  7. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  8. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  9. 2024Optimal ablation for interpretabilitycited by 21
  10. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20
  11. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 17
  12. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  13. 2024Has this Fact been Edited? Detecting Knowledge Edits in Language Modelscited by 14
  14. 2024Parenting: Optimizing Knowledge Selection of Retrieval-Augmented Language Models with Parameter Decoupling and Tailored Tuningcited by 14
  15. 2025Implicit Reasoning in Transformers is Reasoning through Shortcutscited by 14
  16. 2025LLMs as a synthesis between symbolic and distributed approaches to languagecited by 13
  17. 2024Identifying Query-Relevant Neurons in Large Language Models for Long-Form Textscited by 12
  18. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
  19. 2024Mitigating Copy Bias in In-Context Learning through Neuron Pruningcited by 11
  20. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  21. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  22. 2025From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignmentcited by 9
  23. 2025Calibration Across Layers: Understanding Calibration Evolution in LLMscited by 9
  24. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  25. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  26. 2024Neuron-based Multifractal Analysis of Neuron Interaction Dynamics in Large Modelscited by 7
  27. 2024Survival of the Fittest Representation: A Case Study with Modular Additioncited by 7
  28. 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretabilitycited by 7
  29. 2024Wasserstein Distances, Neuronal Entanglement, and Sparsitycited by 6
  30. 2025Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Modelscited by 6
  31. 2025Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learningcited by 6
  32. 2026Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Modelscited by 6
  33. 2026The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Researchcited by 6
  34. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  35. 2025Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inferencecited by 5
  36. 2025Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attributioncited by 5
  37. 2026Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernelcited by 5
  38. 2025On Relation-Specific Neurons in Large Language Modelscited by 4

50 most cited of 100 All 100 (JSON)