# What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes

arXiv: https://arxiv.org/abs/2312.03096 · HTML: https://mlsanity.com/paper/2312.03096/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 18 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 4 of 4)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process](https://mlsanity.com/paper/1904.09080/) (2019)

## Built on by (most cited first, 19 of 19)

- [Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use](https://mlsanity.com/paper/2605.02964/) (2026)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)
- [Towards Uncovering How Large Language Model Works: An Explainability Perspective](https://mlsanity.com/paper/2402.10688/) (2024)
- [On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy](https://mlsanity.com/paper/2506.15963/) (2025)
- [Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness](https://mlsanity.com/paper/2410.21331/) (2024)
- [Towards Combinatorial Interpretability of Neural Computation](https://mlsanity.com/paper/2504.08842/) (2025)
- [Adversarial Attacks Leverage Interference Between Features in Superposition](https://mlsanity.com/paper/2510.11709/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)
- [Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution](https://mlsanity.com/paper/2502.06809/) (2025)
- [Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence](https://mlsanity.com/paper/2505.11611/) (2025)
- [On the creation of narrow AI: hierarchy and nonlocality of neural network skills](https://mlsanity.com/paper/2505.15811/) (2025)
- [Evaluating Sparse Autoencoders for Monosemantic Representation](https://mlsanity.com/paper/2508.15094/) (2025)
- [Spectral Superposition: A Theory of Feature Geometry](https://mlsanity.com/paper/2602.02224/) (2026)
- [Expand Neurons, Not Parameters](https://mlsanity.com/paper/2510.04500/) (2025)
- [SAFR: Neuron Redistribution for Interpretability](https://mlsanity.com/paper/2501.16374/) (2025)
- [Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks](https://mlsanity.com/paper/2503.06181/) (2025)
- [Negative Pre-activations Differentiate Syntax](https://mlsanity.com/paper/2509.24198/) (2025)
- [Interpreting ResNet-based CLIP via Neuron-Attention Decomposition](https://mlsanity.com/paper/2509.19943/) (2025)
- [Interference Beyond Geometry in Concept Extraction](https://mlsanity.com/paper/2609.35351/) (2026)

## Abstract (arXiv)

Polysemantic neurons -- neurons that activate for a set of unrelated features -- have been seen as a significant obstacle towards interpretability of task-optimized deep networks, with implications for AI safety. The classic origin story of polysemanticity is that the data contains more ``features" than neurons, such that learning to perform a task forces the network to co-allocate multiple unrelated features to the same neuron, endangering our ability to understand networks' internal processing. In this work, we present a second and non-mutually exclusive origin story of polysemanticity. We show that polysemanticity can arise incidentally, even when there are ample neurons to represent all features in the data, a phenomenon we term textitincidental polysemanticity. Using a combination of theory and experiments, we show that incidental polysemanticity can arise due to multiple reasons including regularization and neural noise; this incidental polysemanticity occurs because random initialization can, by chance alone, initially assign multiple features to the same neuron, and the training dynamics then strengthen such overlap. Our paper concludes by calling for further research quantifying the performance-polysemanticity tradeoff in task-optimized deep neural networks to better understand to what extent polysemanticity is avoidable.