Data to 7 Oct 2026

Polysemanticity and Capacity in Neural Networks

arXiv 2210.01892 ↗First posted 4 Oct 2022cs.NE cs.AI cs.LGcited by 74

Not on the map. Why?

Abstract (arXiv)

Individual neurons in neural networks often represent a mixture of unrelated features. This phenomenon, called polysemanticity, can make interpreting neural networks more difficult and so we aim to understand its causes. We propose doing so through the lens of feature capacity, which is the fractional dimension each feature consumes in the embedding space. We show that in a toy model the optimal capacity allocation tends to monosemantically represent the most important features, polysemantically represent less important features (in proportion to their impact on the loss), and entirely ignore the least important features. Polysemanticity is more prevalent when the inputs have higher kurtosis or sparsity and more prevalent in some architectures than others. Given an optimal allocation of capacity, we go on to study the geometry of the embedding space. We find a block-semi-orthogonal structure, with differing block sizes in different models, highlighting the impact of model architecture on the interpretability of its neurons.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 1 of its 5 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

1 of 5 references.

References · 1

2022Toy Models of Superpositioncited by 928

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 9; 2025: 31; 2026: 23.

74 papers.

Sort:

By area:

2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
2024Mathematical Models of Computation in Superpositioncited by 34
2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 32
2025Mechanistic understanding and validation of large AI models with SemanticLenscited by 29
2023Disentangling Neuron Representations with Concept Vectorscited by 28

38 more

2025A Closer Look at Multimodal Representation Collapsecited by 25
2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
2024On the Complexity of Neural Computation in Superpositioncited by 21
2022Engineering Monosemanticity in Toy Modelscited by 13
2025On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedycited by 12
2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 11
2025How Causal Abstraction Underpins Computational Explanationcited by 10
2025Towards Combinatorial Interpretability of Neural Computationcited by 9
2025Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMscited by 9
2025Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Frameworkcited by 9
2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 9
2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
2025Memorization Sinks: Isolating Memorization during LLM Trainingcited by 7
2025Adversarial Attacks Leverage Interference Between Features in Superpositioncited by 7
2023Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shiftscited by 6
2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
2025Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial Vulnerabilitycited by 6
2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinkingcited by 6
2025Emergent Symbol-like Number Variables in Artificial Neural Networkscited by 5
2025Self-Regularization with Sparse Autoencoders for Controllable LLM-based Classificationcited by 4
2023Measuring Feature Sparsity in Language Modelscited by 3
2025Localizing Persona Representations in LLMscited by 3
2025Evaluating Sparse Autoencoders for Monosemantic Representationcited by 3
2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
2025Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Modelcited by 3
2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 3
2023SPADE: Sparsity-Guided Debugging for Deep Neural Networkscited by 2
2023Generating Interpretable Networks using Hypernetworkscited by 2
2024Causal Abstraction in Model Interpretability: A Compact Surveycited by 2
2025QPM: Discrete Optimization for Globally Interpretable Image Classificationcited by 2
2025Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitaskingcited by 2
2025Expand Neurons, Not Parameterscited by 2
2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
2025Sparsity and Superposition in Mixture of Expertscited by 2
2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
2026Towards a mathematical theory of superpositioncited by 2
2024Parameter Choice and Neuro-Symbolic Approaches for Deep Domain-Invariant Learningcited by 1

50 most cited of 74 All 74 (JSON)