Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens
Not on the map. Why?
Abstract (arXiv)
Activation steering presupposes that task-relevant behaviors correspond to linear directions in activation space -- directions that should both steer the model and be readable along the unembedding. Function vectors (FVs), extracted as mean differences across ICL demonstrations, are the canonical test case; the prediction: steering and decoding succeed or fail together. Across 12 tasks, 6 models from 3 families, and 4,032 directed cross-template pairs, we find the opposite. FV steering routinely succeeds where the logit lens cannot decode the correct answer at any intermediate layer, while the converse -- decodable without steerable -- is nearly empty (3 of 72). The gap is not representational dialect. A diagonal tuned lens closes 1 of 14 steerable-not-decodable cases; a 2-layer MLP probe with a Hewitt & Liang control closes 5 of 10 via nonlinearly encoded structure but leaves 5 invisible to every decoder tested. Even at > 0.90 steering accuracy, projecting the FV through the unembedding yields incoherent token distributions: FVs encode computational instructions, not answer directions. A model-family asymmetry sharpens the picture. Mistral FVs rewrite intermediate representations, while Llama and Gemma FVs steer the final output without leaving a logit-lens-visible trace, corroborated by three signals (post-steering deltas, activation-patching recovery, FV norm-transfer correlations). A previously reported negative cosine-transfer correlation dissolves at scale, adding at most Delta R2 = 0.011 beyond task identity. These results decompose the linear representation hypothesis into linear decodability and linear steerability and show they come apart opposite to intuition, with implications for safety monitoring: vocabulary-projection tools are blind to FV-style interventions on widely deployed model families.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 3 Apr 2026.
Builds on
32 of 44 references.
References · 32
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
20 more
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
- 2023In-Context Learning Creates Task Vectorscited by 302
- 2023Function Vectors in Large Language Modelscited by 292
- 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
- 2023Linear Representations of Sentiment in Large Language Modelscited by 157
- 2022Analyzing Transformers in Embedding Spacecited by 131
- 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
- 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
- 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
- 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
- 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
- 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 39
- 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
- 2025Reasoning-Finetuning Repurposes Latent Representations in Base Modelscited by 23
- 2025Steering LLM Reasoning Through Bias-Only Adaptationcited by 16
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimescited by 2
- 2026Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languagescited by 1
- 2026Training, Reading, and Editing Legible Transformerscited by 0
- 2026Transformation Laws in Neural Representations: Structure, Realisability, and Constructioncited by 0
- 2026Adaptive Multi-Value Control in LLMs via Causal Activation Steeringcited by 0
- 2026Adaptive Multi-Value Control in LLMs via Causal Activation Steering24 Sep 2026
- 2026Transformation Laws in Neural Representations: Structure, Realisability, and Construction16 Sep 2026
- 2026Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages4 Aug 2026
- 2026Training, Reading, and Editing Legible Transformers9 Jul 2026
- 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes7 May 2026