mlsanityData to 7 Oct 2026

Disentangling MLP Neuron Weights in Vocabulary Space

arXiv 2604.06005 ↗First posted 7 Apr 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Interpreting the information encoded in language model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a data-free method requiring no forward passes that disentangles MLP neurons directly in weight space. Our approach relies on a key statistical observation: neurons that encode coherent, monosemantic concepts exhibit high kurtosis when projected onto the model's vocabulary. By optimizing rotations of neuron weights to maximize their vocabulary-space kurtosis, our method recovers sparse, interpretable directions which we name vocabulary channels. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it demonstrate that ROTATE consistently recovers vocabulary channels that are faithful to the neuron's behavior; ablating individual channels selectively disables corresponding input activations or the promotion of specific concepts. Moreover, aggregating channel-level descriptions yields comprehensive neuron descriptions that outperform optimized activation-based baselines by 2-3x in head-to-head comparisons. By providing a data-free decomposition of neuron weights, ROTATE offers a scalable, fine-grained building block for interpreting language models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

43 of 59 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2022Toy Models of Superpositioncited by 928
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  10. 2024Scaling and evaluating sparse autoencoderscited by 530
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  12. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
31 more
  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  2. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  3. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  4. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  5. 2025Open Problems in Mechanistic Interpretabilitycited by 173
  6. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  7. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  8. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  9. 2022Analyzing Transformers in Embedding Spacecited by 131
  10. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  11. 2024Universal Neurons in GPT2 Language Modelscited by 100
  12. 2021An Interpretability Illusion for BERTcited by 94
  13. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  14. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  15. 2022Polysemanticity and Capacity in Neural Networkscited by 74
  16. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  17. 2024Attention Heads of Large Language Models: A Surveycited by 65
  18. 2024A Multimodal Automated Interpretability Agentcited by 62
  19. 2024Confidence Regulation Neurons in Language Modelscited by 57
  20. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  21. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  22. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  23. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  24. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  25. 2024Bilinear MLPs enable weight-based mechanistic interpretabilitycited by 28
  26. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
  27. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  28. 2024Inferring Functionality of Attention Heads from their Parameterscited by 17
  29. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  30. 2025Towards Combinatorial Interpretability of Neural Computationcited by 9
  31. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Neuron Populations Exhibit Divergent Selectivity with Scalecited by 0