# Compressed Computation is (probably) not Computation in Superposition

arXiv: https://arxiv.org/abs/2606.14673 · HTML: https://mlsanity.com/paper/2606.14673/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 3 of its 7 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 3 of 3)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition](https://mlsanity.com/paper/2501.14926/) (2025)
- [On the Complexity of Neural Computation in Superposition](https://mlsanity.com/paper/2409.15318/) (2024)

## Built on by (most cited first, 2 of 2)

- [Compressed Computation under L4 Loss is likely Computation in Superposition](https://mlsanity.com/paper/2607.04800/) (2026)
- [The Ball and the Box: Two Geometries of Computation in Superposition](https://mlsanity.com/paper/2610.11744/) (2026)

## Abstract (arXiv)

We study whether the Compressed Computation (CC) toy model (Braun et al., 2025) is an instance of computation in superposition. The CC model appears to compute 100 ReLU functions with just 50 neurons, achieving a better loss than expected from only representing 50 ReLU functions. We show that the model mixes inputs via its noisy residual stream, corresponding to an unintended mixing matrix in the labels. Splitting the training objective into the ReLU term and the mixing term, we find that performance gains scale with the magnitude of the mixing matrix and vanish when the matrix is removed. The learned neuron directions concentrate in the subspace associated with the top 50 eigenvalues of the mixing matrix, suggesting that the mixing term governs the solution. Finally, a semi-non-negative matrix factorization (SNMF) baseline derived solely from the mixing matrix reproduces the qualitative loss profile and improves on prior baselines, though it does not match the trained model. These results suggest CC is not a suitable toy model of computation in superposition.