When Does Activation Steering Change What a Model Computes From?
Not on the map. Why?
Abstract (arXiv)
Activation steering can reliably change an agent's output by modifying its internal activations. Yet arriving at the same answer need not involve the same computation: behavioral equivalence does not imply mechanistic equivalence. We test whether an activation edit changes the state used in subsequent computation or biases that computation toward the desired output. In a controlled state-tracking task, trace supervision creates an editable register: changing its internal value makes the model apply the next operation to the edited state. We next ask whether widely used mean activation steering admits a similar interpretation: does steering recreate the internal configuration the model naturally uses when performing a task? If so, transplanting that activation should be effective, and steering should remain effective at the scale of the natural source-to-target activation change. Neither prediction holds in the two selected model-task settings. Replacing the activation at one layer produces less than 2% of the target-answer margin gain from patching through all remaining layers, while natural-scale steering is similarly ineffective. The steering vectors have norms 77 and 26 times the median natural change in Qwen and Llama, producing 54% and 82% of the reference effect. Thus, a successful steering intervention need not reproduce the natural target activation at the intervention layer. More generally, an intervention should be interpreted as changing the computational state only when a later computation uses the edited value according to the semantics of that state.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 28 Jun 2026.
Builds on
24 of 30 references.
References · 24
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Editing Models with Task Arithmeticcited by 1,076
- 2022In-context Learning and Induction Headscited by 930
- 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
- 2021Causal Abstractions of Neural Networkscited by 401
- 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
12 more
- 2023Function Vectors in Large Language Modelscited by 292
- 2022Transformers Learn Shortcuts to Automatacited by 260
- 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
- 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
- 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
- 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
- 2025Language Models Use Trigonometry to Do Additioncited by 68
- 2025Language Models use Lookbacks to Track Beliefscited by 36
- 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
- 2025Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Trackingcited by 17
- 2026Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Conceptscited by 9
- 2026On the Non-Identifiability of Steering Vectors in Large Language Modelscited by 8
Built on by
Citing papers per year: 2026: 2.
2 papers.