# Understanding polysemanticity in neural networks through coding theory

arXiv: https://arxiv.org/abs/2401.17975 · HTML: https://mlsanity.com/paper/2401.17975/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients](https://mlsanity.com/paper/1711.09404/) (2017)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks](https://mlsanity.com/paper/1801.03454/) (2018)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Adversarial Robustness on In- and Out-Distribution Improves Explainability](https://mlsanity.com/paper/2003.09461/) (2020)
- [Planting Undetectable Backdoors in Machine Learning Models](https://mlsanity.com/paper/2204.06974/) (2022)
- [Scaleable input gradient regularization for adversarial robustness](https://mlsanity.com/paper/1905.11468/) (2019)
- [Issues in evaluating semantic spaces using word analogies](https://mlsanity.com/paper/1606.07736/) (2016)
- [An Information Theoretic Interpretation to Deep Neural Networks](https://mlsanity.com/paper/1905.06600/) (2019)
- [Investigating Power laws in Deep Representation Learning](https://mlsanity.com/paper/2202.05808/) (2022)
- [Towards Benchmarking Explainable Artificial Intelligence Methods](https://mlsanity.com/paper/2208.12120/) (2022)

## Built on by (most cited first, 17 of 17)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Pruning for Robust Concept Erasing in Diffusion Models](https://mlsanity.com/paper/2405.16534/) (2024)
- [Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness](https://mlsanity.com/paper/2410.21331/) (2024)
- [Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework](https://mlsanity.com/paper/2506.15538/) (2025)
- [CoE: Chain-of-Explanation via Automatic Visual Concept Circuit Description and Polysemanticity Quantification](https://mlsanity.com/paper/2503.15234/) (2025)
- [Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders](https://mlsanity.com/paper/2510.23802/) (2025)
- [Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial Vulnerability](https://mlsanity.com/paper/2512.13568/) (2025)
- [Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution](https://mlsanity.com/paper/2502.06809/) (2025)
- [Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence](https://mlsanity.com/paper/2505.11611/) (2025)
- [Evaluating Sparse Autoencoders for Monosemantic Representation](https://mlsanity.com/paper/2508.15094/) (2025)
- [SAFR: Neuron Redistribution for Interpretability](https://mlsanity.com/paper/2501.16374/) (2025)
- [Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders](https://mlsanity.com/paper/2507.15227/) (2025)
- [Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs](https://mlsanity.com/paper/2509.04655/) (2025)
- [Representational Curvature Modulates Behavioral Uncertainty in Large Language Models](https://mlsanity.com/paper/2604.23985/) (2026)
- [fmxcoders: Factorized Masked Crosscoders for Cross-Layer Feature Discovery](https://mlsanity.com/paper/2605.09438/) (2026)
- [SPICE: Simple Polysemantic Feature Interpretation via Clustering-based Explanation](https://mlsanity.com/paper/2609.13198/) (2026)
- [RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing](https://mlsanity.com/paper/2610.11775/) (2026)

## Abstract (arXiv)

Despite substantial efforts, neural network interpretability remains an elusive goal, with previous research failing to provide succinct explanations of most single neurons' impact on the network output. This limitation is due to the polysemantic nature of most neurons, whereby a given neuron is involved in multiple unrelated network states, complicating the interpretation of that neuron. In this paper, we apply tools developed in neuroscience and information theory to propose both a novel practical approach to network interpretability and theoretical insights into polysemanticity and the density of codes. We infer levels of redundancy in the network's code by inspecting the eigenspectrum of the activation's covariance matrix. Furthermore, we show how random projections can reveal whether a network exhibits a smooth or non-differentiable code and hence how interpretable the code is. This same framework explains the advantages of polysemantic neurons to learning performance and explains trends found in recent results by Elhage et al.~(2022). Our approach advances the pursuit of interpretability in neural networks, providing insights into their underlying structure and suggesting new avenues for circuit-level interpretability.