# On Implications of Scaling Laws on Feature Superposition

arXiv: https://arxiv.org/abs/2407.01459 · HTML: https://mlsanity.com/paper/2407.01459/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 4 of 4)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)

## Built on by (most cited first, 1 of 1)

- [SAFR: Neuron Redistribution for Interpretability](https://mlsanity.com/paper/2501.16374/) (2025)

## Abstract (arXiv)

Using results from scaling laws, this theoretical note argues that the following two statements cannot be simultaneously true: 1. Superposition hypothesis where sparse features are linearly represented across a layer is a complete theory of feature representation. 2. Features are universal, meaning two models trained on the same data and achieving equal performance will learn identical features.