# Compressed Computation under L4 Loss is likely Computation in Superposition

arXiv: https://arxiv.org/abs/2607.04800 · HTML: https://mlsanity.com/paper/2607.04800/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 6 of 6)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Mathematical Models of Computation in Superposition](https://mlsanity.com/paper/2408.05451/) (2024)
- [Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition](https://mlsanity.com/paper/2501.14926/) (2025)
- [On the Complexity of Neural Computation in Superposition](https://mlsanity.com/paper/2409.15318/) (2024)
- [Stochastic Parameter Decomposition](https://mlsanity.com/paper/2506.20790/) (2025)
- [Compressed Computation is (probably) not Computation in Superposition](https://mlsanity.com/paper/2606.14673/) (2026)

## Abstract (arXiv)

Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an L4 loss (the mean fourth power of the error), rather than the usual L2, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.