# Evidence for feature-specific error correction in LLMs

arXiv: https://arxiv.org/abs/2606.24964 · HTML: https://mlsanity.com/paper/2606.24964/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [Mathematical Models of Computation in Superposition](https://mlsanity.com/paper/2408.05451/) (2024)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [On the Complexity of Neural Computation in Superposition](https://mlsanity.com/paper/2409.15318/) (2024)
- [Characterizing stable regions in the residual stream of LLMs](https://mlsanity.com/paper/2409.17113/) (2024)

## Built on by (most cited first, 1 of 1)

- [Language Model Activations Inhabit Privileged Error-Correcting Basins](https://mlsanity.com/paper/2610.04183/) (2026)

## Abstract (arXiv)

Understanding the features of large language models (LLMs) is a central goal of interpretability. LLMs are commonly assumed to use superposition to represent more features than they have dimensions. They may not only represent features in superposition but also perform computation in superposition. Theory predicts that computing in superposition requires error correction that privileges feature directions over generic ones, but this prediction has not been tested empirically. We propose an empirical test of error correction in LLMs based on activation perturbations. Perturbing residual-stream activations, we find that they are robust to small perturbations--forming activation plateaus consistent with error correction--but less robust along candidate feature directions ("pure" directions, constructed from contrastive prompt pairs) than along mixtures of two such directions, indicating that the pure directions are privileged. We quantify this privilegedness by modeling the perturbation effect as a function of the Lp-norm of its decomposition into feature components. For p=2 the response is a quadratic form with at most as many nonzero eigenvalues as the residual-stream dimension, which cannot privilege the many feature directions superposition requires. p>2 lifts this constraint and is consistent with feature-specific error correction. We find p>2 for contrastive, MELBO, and SAE-decoder directions, and p≈2 for random and PCA directions (controls). These results replicate across Gemma-2-9B, Qwen3-1.7B, Llama-3.1-8B, Mistral-7B-v0.3, Aya-Expanse-8B, and Yi-1.5-9B. We further validate our method on a toy model of error correction with known ground-truth features, recovering p>2 for true feature directions, degrading toward 2 as we rotate away from them.