Steering grids for sparse-autoencoder features: when a top-context label names an activation regime rather than a causal axis
On the map: Mechanistic interpretability → SAE features and their applications
Abstract (arXiv)
The standard protocol for interpreting sparse-autoencoder (SAE) features labels each feature from its top-activating contexts and validates the label by steering that single feature at a typical magnitude. We argue that this inspects one cell of a larger steering grid, steering condition (single feature, joint feature set, matched random direction) crossed with steering coefficient, and show that other cells carry information that changes the label. On Qwen3-1.7B-Instruct and Gemma-2-2B-it, with the matched-geometry control extended to Llama-3.1-8B-Instruct: (1) features labelled AI self-disclaimer from their top contexts switch to a second surface form under steering, a contemplative voice on Qwen, a collective we-voice on Gemma, so the label names an activation regime, not the causal axis; two anchor features separate genuine mode switches from monotonic response and from breakdown. (2) Three near-orthogonal features that are individually substitutable are jointly necessary for grounded composition: joint suppression collapses unrelated control tasks into placeholder text that single-feature suppression at the same coefficient leaves intact. (3) A matched-geometry random-direction control shows the collapse is direction-pattern-dependent, not magnitude-dependent: at the same residual-stream distortion, feature directions damage unrelated tasks where magnitude-matched random directions do not, with non-overlapping 95% confidence intervals on all three models, including the one SAE trained on the model it is applied to.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 30 of its 35 references that are in the map. First posted on arXiv 4 May 2026.
Builds on
30 of 35 references.
Closest ancestors · 10
Same direction, newest first
- 2025SAEs Are Good for Steering -- If You Select the Right Features26 May
- 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability12 Mar
- 2025FADE: Why Bad Descriptions Happen to Good Features24 Feb
- 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing23 Feb
- 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders28 Jan
- 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
- 2024Automatically Interpreting Millions of Features in Large Language Models17 Oct
- 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
- 2024Scaling and evaluating sparse autoencoders6 Jun
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep
Other references · 20
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2022Toy Models of Superpositioncited by 971
- 2023Steering Language Models With Activation Engineeringcited by 922
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
- 2019Neural Text Generation with Unlikelihood Trainingcited by 503
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
- 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
- 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
8 more
- 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
- 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
- 2022Language Models as Agent Modelscited by 132
- 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 88
- 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 45
- 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 41
- 2024Towards Reliable Evaluation of Behavior Steering Interventions in LLMscited by 36
- 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 21
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Steering Large Language Model Activations in Sparse Spaces
- 2026Pre-Intervention Prediction of Sparse Autoencoder Steering Side Effects
- 2025Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering
- 2026When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic Interpretability
- 2026From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
- 2024Can sparse autoencoders be used to decompose and interpret steering vectors?
- 2026Disentangling Steering Vectors
- 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering