mlsanityData to 7 Oct 2026

Toy Models of Superposition

arXiv 2209.10652 ↗First posted 21 Sep 2022cs.LGcited by 928

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in "superposition." We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.

Key facts. Cited by 928 papers in the map (data through 7 Oct 2026). Builds on 3 of its 5 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

3 of 5 references.

References · 3

  1. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  2. 2016Towards the Science of Security and Privacy in Machine Learningcited by 89
  3. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 65; 2024: 150; 2025: 273; 2026: 434.

928 papers.

Sort:

By area:

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2023Large Language Modelscited by 774
  4. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  7. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  9. 2024Scaling and evaluating sparse autoencoderscited by 530
  10. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  11. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  12. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
38 more
  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  2. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  3. 2023Language Models Represent Space and Timecited by 306
  4. 2023Large Language Model Alignment: A Surveycited by 241
  5. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  6. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  7. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  8. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  9. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  10. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  11. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  12. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  13. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  14. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
  15. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  16. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  17. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  18. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
  19. 2024KAN 2.0: Kolmogorov-Arnold Networks Meet Sciencecited by 116
  20. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  21. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  22. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  23. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  24. 2023Explaining grokking through circuit efficiencycited by 105
  25. 2024Universal Neurons in GPT2 Language Modelscited by 100
  26. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  27. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  28. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
  29. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  30. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  31. 2024Truth is Universal: Robust Detection of Lies in LLMscited by 78
  32. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  33. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  34. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  35. 2022Polysemanticity and Capacity in Neural Networkscited by 74
  36. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  37. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  38. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70

50 most cited of 928 All 928 (JSON)