mlsanityData to 7 Oct 2026

Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens

arXiv 2604.02608 ↗First posted 3 Apr 2026cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Activation steering presupposes that task-relevant behaviors correspond to linear directions in activation space -- directions that should both steer the model and be readable along the unembedding. Function vectors (FVs), extracted as mean differences across ICL demonstrations, are the canonical test case; the prediction: steering and decoding succeed or fail together. Across 12 tasks, 6 models from 3 families, and 4,032 directed cross-template pairs, we find the opposite. FV steering routinely succeeds where the logit lens cannot decode the correct answer at any intermediate layer, while the converse -- decodable without steerable -- is nearly empty (3 of 72). The gap is not representational dialect. A diagonal tuned lens closes 1 of 14 steerable-not-decodable cases; a 2-layer MLP probe with a Hewitt & Liang control closes 5 of 10 via nonlinearly encoded structure but leaves 5 invisible to every decoder tested. Even at > 0.90 steering accuracy, projecting the FV through the unembedding yields incoherent token distributions: FVs encode computational instructions, not answer directions. A model-family asymmetry sharpens the picture. Mistral FVs rewrite intermediate representations, while Llama and Gemma FVs steer the final output without leaving a logit-lens-visible trace, corroborated by three signals (post-steering deltas, activation-patching recovery, FV norm-transfer correlations). A previously reported negative cosine-transfer correlation dissolves at scale, adding at most Delta R2 = 0.011 beyond task identity. These results decompose the linear representation hypothesis into linear decodability and linear steerability and show they come apart opposite to intuition, with implications for safety monitoring: vocabulary-projection tools are blind to FV-style interventions on widely deployed model families.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

32 of 44 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  10. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  11. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
20 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  3. 2019Designing and Interpreting Probes with Control Taskscited by 608
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  6. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  7. 2023In-Context Learning Creates Task Vectorscited by 302
  8. 2023Function Vectors in Large Language Modelscited by 292
  9. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  10. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  11. 2022Analyzing Transformers in Embedding Spacecited by 131
  12. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  13. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  14. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  15. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  16. 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
  17. 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 39
  18. 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
  19. 2025Reasoning-Finetuning Repurposes Latent Representations in Base Modelscited by 23
  20. 2025Steering LLM Reasoning Through Bias-Only Adaptationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.