# Spectral Superposition: A Theory of Feature Geometry

arXiv: https://arxiv.org/abs/2602.02224 · HTML: https://mlsanity.com/paper/2602.02224/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Multi-Attribute Steering of Language Models via Targeted Intervention](https://mlsanity.com/paper/2502.12446/) (2025)
- [When Models Manipulate Manifolds: The Geometry of a Counting Task](https://mlsanity.com/paper/2601.04480/) (2026)
- [PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits](https://mlsanity.com/paper/2404.06453/) (2024)
- [On the Complexity of Neural Computation in Superposition](https://mlsanity.com/paper/2409.15318/) (2024)
- [What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes](https://mlsanity.com/paper/2312.03096/) (2023)
- [Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal](https://mlsanity.com/paper/2509.09708/) (2025)
- [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025)
- [Deep sequence models tend to memorize geometrically; it is unclear why](https://mlsanity.com/paper/2510.26745/) (2025)
- [Disentangling Polysemantic Channels in Convolutional Neural Networks](https://mlsanity.com/paper/2504.12939/) (2025)
- [Superposition in Graph Neural Networks](https://mlsanity.com/paper/2509.00928/) (2025)
- [Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models](https://mlsanity.com/paper/2507.15798/) (2025)

## Built on by (most cited first, 4 of 4)

- [Towards a mathematical theory of superposition](https://mlsanity.com/paper/2608.27540/) (2026)
- [Interference Beyond Geometry in Concept Extraction](https://mlsanity.com/paper/2609.35351/) (2026)
- [Emergent phases of superposition: from partial to full representation](https://mlsanity.com/paper/2609.36455/) (2026)
- [Component and Dimension Sparsity in Transformer Refusal Mechanisms](https://mlsanity.com/paper/2610.06903/) (2026)

## Abstract (arXiv)

Neural networks represent more features than they have dimensions via superposition, forcing features to share representational space. Current methods decompose activations into sparse linear features but discard geometric structure. We develop a theory for studying the geometric structre of features by analyzing the spectra (eigenvalues, eigenspaces, etc.) of weight derived matrices. In particular, we introduce the frame operator F = WW^top, which gives us a spectral measure that describes how each feature allocates norm across eigenspaces. While previous tools could describe the pairwise interactions between features, spectral methods capture the global geometry (``how do all features interact?''). In toy models of superposition, we use this theory to prove that capacity saturation forces spectral localization: features collapse onto single eigenspaces, organize into tight frames, and admit discrete classification via association schemes, classifying all geometries from prior work (simplices, polygons, antiprisms). The spectral measure formalism applies to arbitrary weight matrices, enabling diagnosis of feature localization beyond toy settings. These results point toward a broader program: applying operator theory to interpretability.