# Towards a mathematical theory of superposition

arXiv: https://arxiv.org/abs/2608.27540 · HTML: https://mlsanity.com/paper/2608.27540/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 48 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 10 of 10)

- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Polysemanticity and Capacity in Neural Networks](https://mlsanity.com/paper/2210.01892/) (2022)
- [Superposition of many models into one](https://mlsanity.com/paper/1902.05522/) (2019)
- [Spectral Superposition: A Theory of Feature Geometry](https://mlsanity.com/paper/2602.02224/) (2026)
- [The Persian Rug: solving toy models of superposition using large-scale symmetries](https://mlsanity.com/paper/2410.12101/) (2024)
- [Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes](https://mlsanity.com/paper/2605.01192/) (2026)

## Built on by (most cited first, 2 of 2)

- [Emergent phases of superposition: from partial to full representation](https://mlsanity.com/paper/2609.36455/) (2026)
- [The Ball and the Box: Two Geometries of Computation in Superposition](https://mlsanity.com/paper/2610.11744/) (2026)

## Abstract (arXiv)

We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector (x) of active features is encoded through an overcomplete dictionary (W), and feature recovery is performed by applying (operatornameReLU(W^top W x+b)) with an appropriate bias vector (b). We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order (d/log n). In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with (n>d+1), we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.