# Emergent phases of superposition: from partial to full representation

arXiv: https://arxiv.org/abs/2609.36455 · HTML: https://mlsanity.com/paper/2609.36455/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Polysemanticity and Capacity in Neural Networks](https://mlsanity.com/paper/2210.01892/) (2022)
- [Simple, Efficient, and Neural Algorithms for Sparse Coding](https://mlsanity.com/paper/1503.00778/) (2015)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition](https://mlsanity.com/paper/2310.06301/) (2023)
- [A note on the sample complexity of the Er-SpUD algorithm by Spielman, Wang and Wright for exact recovery of sparsely used dictionaries](https://mlsanity.com/paper/1601.02049/) (2016)
- [Inverse Depth Scaling From Most Layers Being Similar](https://mlsanity.com/paper/2602.05970/) (2026)
- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Spectral Superposition: A Theory of Feature Geometry](https://mlsanity.com/paper/2602.02224/) (2026)
- [The Persian Rug: solving toy models of superposition using large-scale symmetries](https://mlsanity.com/paper/2410.12101/) (2024)
- [Geometric Limits of Knowledge Distillation: A Minimum-Width Theorem via Superposition Theory](https://mlsanity.com/paper/2604.04037/) (2026)
- [Towards a mathematical theory of superposition](https://mlsanity.com/paper/2608.27540/) (2026)

## Abstract (arXiv)

Large language models are thought to represent features by vectors in a hidden space of dimension given by the model's width. Superposition, in which more features are represented than the width by letting representation vectors overlap, is a leading account of how representation vectors are organized. However, how model width and data statistics determine the configuration of representation vectors and the resulting loss when the number of features and the width are large remains less understood. Here we show, in Anthropic's toy model of superposition, that increasing the width drives a continuous phase transition from a partial-representation phase, where only a subset of features receives appreciable representation vectors while the rest vanish, to a full-representation phase, where every feature is represented. Our theory via a partial random projection approximation predicts, and experiments confirm, that the critical width grows linearly with the number of active features up to a logarithmic factor. The loss scaling changes across the transition: below the critical width, the loss grows linearly with the number of active features and depends weakly on the width in a form set by data statistics; above it, the loss grows approximately quadratically with the number of active features and decays inversely with the width. Non-uniform firing probabilities delay the transition and lower the loss, as more frequent features occupy more space. Our results provide an account of how model width and data statistics jointly shape representations and loss, a step toward understanding representation scaling in large models.