# Decomposing The Dark Matter of Sparse Autoencoders

arXiv: https://arxiv.org/abs/2410.14670 · HTML: https://mlsanity.com/paper/2410.14670/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models](https://mlsanity.com/paper/2408.00113/) (2024)
- [Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations](https://mlsanity.com/paper/2408.10920/) (2024)
- [Evaluating Synthetic Activations composed of SAE Latents in GPT-2](https://mlsanity.com/paper/2409.15019/) (2024)
- [Discriminative reconstruction via simultaneous dense and sparse coding](https://mlsanity.com/paper/2006.09534/) (2020)
- [Investigating Sensitive Directions in GPT-2: An Improved Baseline and Comparative Analysis of SAEs](https://mlsanity.com/paper/2410.12555/) (2024)

## Built on by (most cited first, 30 of 45)

- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning](https://mlsanity.com/paper/2504.02922/) (2025)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [Transferring Linear Features Across Language Models With Model Stitching](https://mlsanity.com/paper/2506.06609/) (2025)
- [Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders](https://mlsanity.com/paper/2505.11756/) (2025)
- [A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning](https://mlsanity.com/paper/2411.04105/) (2024)
- [Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders](https://mlsanity.com/paper/2508.16560/) (2025)
- [Understanding sparse autoencoder scaling in the presence of feature manifolds](https://mlsanity.com/paper/2509.02565/) (2025)
- [Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition](https://mlsanity.com/paper/2504.20938/) (2025)
- [AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features](https://mlsanity.com/paper/2510.00404/) (2025)
- [Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality](https://mlsanity.com/paper/2503.24277/) (2025)
- [Causality is Key for Interpretability Claims to Generalise](https://mlsanity.com/paper/2602.16698/) (2026)
- [Low-Rank Adapting Models for Sparse Autoencoders](https://mlsanity.com/paper/2501.19406/) (2025)
- [Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition](https://mlsanity.com/paper/2504.00194/) (2025)
- [Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders](https://mlsanity.com/paper/2505.21364/) (2025)
- [Feature Identification via the Empirical NTK](https://mlsanity.com/paper/2510.00468/) (2025)
- [Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders](https://mlsanity.com/paper/2508.09363/) (2025)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [Mechanistically Eliciting Latent Behaviors in Language Models](https://mlsanity.com/paper/2606.29604/) (2026)
- [Representation Learning on a Random Lattice](https://mlsanity.com/paper/2504.20197/) (2025)
- [Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering](https://mlsanity.com/paper/2506.01247/) (2025)
- [Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features](https://mlsanity.com/paper/2602.10437/) (2026)
- [Geometric Routing Enables Causal Expert Control in Mixture of Experts](https://mlsanity.com/paper/2604.14434/) (2026)
- [Feature Integration Spaces: Joint Training Reveals Dual Encoding in Neural Network Representations](https://mlsanity.com/paper/2507.00269/) (2025)
- [Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone](https://mlsanity.com/paper/2509.10809/) (2025)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are a promising technique for decomposing language model activations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in "dark matter": unexplained variance in activations. This work investigates dark matter as an object of study in its own right. Surprisingly, we find that much of SAE dark matter -- about half of the error vector itself and >90% of its norm -- can be linearly predicted from the initial activation vector. Additionally, we find that the scaling behavior of SAE error norms at a per token level is remarkably predictable: larger SAEs mostly struggle to reconstruct the same contexts as smaller SAEs. We build on the linear representation hypothesis to propose models of activations that might lead to these observations. These insights imply that the part of the SAE error vector that cannot be linearly predicted ("nonlinear" error) might be fundamentally different from the linearly predictable component. To validate this hypothesis, we empirically analyze nonlinear SAE error and show that 1) it contains fewer not yet learned features, 2) SAEs trained on it are quantitatively worse, and 3) it is responsible for a proportional amount of the downstream increase in cross entropy loss when SAE activations are inserted into the model. Finally, we examine two methods to reduce nonlinear SAE error: inference time gradient pursuit, which leads to a very slight decrease in nonlinear error, and linear transformations from earlier layer SAE outputs, which leads to a larger reduction.