mlsanityData to 14 Oct 2026

Finding Neurons in a Haystack: Case Studies with Sparse Probing

arXiv 2305.01610 ↗First posted 2 May 2023cs.LG cs.AIcited by 334

Not on the map. Why?

Cited by 107 in Mechanistic interpretability, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train k-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of k we study the sparsity of learned representations and how this varies with model scale. With k=1, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.

Key facts. Cited by 334 papers in the map (data through 14 Oct 2026). Builds on 71 of its 107 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

71 of 107 references.

References · 71

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2016Layer Normalizationcited by 5,669
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
38 more
  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  2. 2018Sanity Checks for Saliency Mapscited by 1,156
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2022Toy Models of Superpositioncited by 971
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  7. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  11. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  12. 2019Designing and Interpreting Probes with Control Taskscited by 640
  13. 2018Disentangling by Factorisingcited by 616
  14. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  15. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  16. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  17. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  18. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  19. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  20. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  21. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  22. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  23. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  24. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  25. 2023Localizing Model Behavior with Path Patchingcited by 195
  26. 2020Compositional Explanations of Neuronscited by 172
  27. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  28. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  29. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  30. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  31. 2023The Quantization Model of Neural Scalingcited by 138
  32. 2022Analyzing Transformers in Embedding Spacecited by 132
  33. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  34. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  35. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  36. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  37. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  38. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 38; 2024: 94; 2025: 111; 2026: 91.

334 papers.

Sort:

By area:

  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2024Scaling and evaluating sparse autoencoderscited by 553
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  4. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  6. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  7. 2023Language Models Represent Space and Timecited by 316
  8. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  9. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  10. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  11. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  12. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
38 more
  1. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 169
  2. 2024The pitfalls of next-token predictioncited by 160
  3. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  4. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  5. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  6. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  7. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  8. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  9. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  10. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
  11. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 124
  12. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  13. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  14. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  15. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  16. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 113
  17. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  18. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  19. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  20. 2024Universal Neurons in GPT2 Language Modelscited by 102
  21. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  22. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 86
  23. 2025Persona Features Control Emergent Misalignmentcited by 86
  24. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  25. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 79
  26. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  27. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  28. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 76
  29. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  30. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  31. 2024A Multimodal Automated Interpretability Agentcited by 63
  32. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
  33. 2023Explaining black box text modules in natural language with language modelscited by 62
  34. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  35. 2024Confidence Regulation Neurons in Language Modelscited by 57
  36. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  37. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  38. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51

50 most cited of 334 All 334 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability
  2. 2026Sparsely gated tiny linear experts
  3. 2025Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
  4. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  5. 2026Interpreting Brain Responses to Language with Sparse Features from Language Models
  6. 2025Evaluating Sparse Autoencoders for Monosemantic Representation
  7. 2025Interpreting Language Models Through Concept Descriptions: A Survey
  8. 2025Weight-sparse transformers have interpretable circuits