# Polysemanticity and Capacity in Neural Networks

arXiv: https://arxiv.org/abs/2210.01892 · HTML: https://mlsanity.com/paper/2210.01892/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 5 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 1 of 1)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)

## Built on by (most cited first, 30 of 77)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Tracr: Compiled Transformers as a Laboratory for Interpretability](https://mlsanity.com/paper/2301.05062/) (2023)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Mathematical Models of Computation in Superposition](https://mlsanity.com/paper/2408.05451/) (2024)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [Mechanistic understanding and validation of large AI models with SemanticLens](https://mlsanity.com/paper/2501.05398/) (2025)
- [Disentangling Neuron Representations with Concept Vectors](https://mlsanity.com/paper/2304.09707/) (2023)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)
- [InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques](https://mlsanity.com/paper/2407.14494/) (2024)
- [Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models](https://mlsanity.com/paper/2502.10835/) (2025)
- [On the Complexity of Neural Computation in Superposition](https://mlsanity.com/paper/2409.15318/) (2024)
- [On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy](https://mlsanity.com/paper/2506.15963/) (2025)
- [Engineering Monosemanticity in Toy Models](https://mlsanity.com/paper/2211.09169/) (2022)
- [Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder](https://mlsanity.com/paper/2502.14050/) (2025)
- [How Causal Abstraction Underpins Computational Explanation](https://mlsanity.com/paper/2508.11214/) (2025)
- [Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework](https://mlsanity.com/paper/2506.15538/) (2025)
- [From Data Statistics to Feature Geometry: How Correlations Shape Superposition](https://mlsanity.com/paper/2603.09972/) (2026)
- [Towards Combinatorial Interpretability of Neural Computation](https://mlsanity.com/paper/2504.08842/) (2025)
- [Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs](https://mlsanity.com/paper/2505.14530/) (2025)
- [Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training](https://mlsanity.com/paper/2509.25758/) (2025)
- [Adversarial Attacks Leverage Interference Between Features in Superposition](https://mlsanity.com/paper/2510.11709/) (2025)
- [Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinking](https://mlsanity.com/paper/2602.20330/) (2026)
- [Memorization Sinks: Isolating Memorization during LLM Training](https://mlsanity.com/paper/2507.09937/) (2025)
- [Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial Vulnerability](https://mlsanity.com/paper/2512.13568/) (2025)

## Abstract (arXiv)

Individual neurons in neural networks often represent a mixture of unrelated features. This phenomenon, called polysemanticity, can make interpreting neural networks more difficult and so we aim to understand its causes. We propose doing so through the lens of feature capacity, which is the fractional dimension each feature consumes in the embedding space. We show that in a toy model the optimal capacity allocation tends to monosemantically represent the most important features, polysemantically represent less important features (in proportion to their impact on the loss), and entirely ignore the least important features. Polysemanticity is more prevalent when the inputs have higher kurtosis or sparsity and more prevalent in some architectures than others. Given an optimal allocation of capacity, we go on to study the geometry of the embedding space. We find a block-semi-orthogonal structure, with differing block sizes in different models, highlighting the impact of model architecture on the interpretability of its neurons.