mlsanityData to 7 Oct 2026

Steered LLM Activations are Non-Surjective

arXiv 2604.09839 ↗First posted 10 Apr 2026cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Activation steering is a popular white-box control technique that modifies model activations to elicit an abstract change in its behavior. It has also become a standard tool in interpretability (e.g., probing truthfulness, or translating activations into human-readable explanations) and safety research (e.g., jailbreakability). However, it is unclear whether steered behavior is realizable by any textual prompt. In this work, we cast this question as a surjectivity problem: for a fixed model, does every steered activation admit a preimage under the model's natural forward pass? Under practical assumptions, we prove that activation steering pushes the residual stream off the manifold of states reachable from discrete prompts. Almost surely, no prompt can reproduce the same internal behavior induced by steering. We also illustrate this finding empirically across three widely used LLMs. Our results establish a formal separation between white-box steerability and black-box prompting. We therefore caution against interpreting the ease and success of activation steering as evidence of prompt-based interpretability or vulnerability, and argue for evaluation protocols that explicitly decouple white-box and black-box interventions.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

46 of 57 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  8. 2023Steering Language Models With Activation Engineeringcited by 824
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  11. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  12. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
34 more
  1. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  2. 2024How to use and interpret activation patchingcited by 191
  3. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  6. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  7. 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 76
  8. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 69
  9. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  10. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  11. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  12. 2025Language Models are Injective and Hence Invertiblecited by 31
  13. 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
  14. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  15. 2025Understanding Refusal in Language Models with Sparse Autoencoderscited by 26
  16. 2025To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Modelscited by 26
  17. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 24
  18. 2025Activation Steering for Chain-of-Thought Compressioncited by 22
  19. 2026Sycophancy Hides Linearly in the Attention Headscited by 21
  20. 2024LiSeCo: Linear Semantic Control for Language Generationcited by 20
  21. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  22. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  23. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  24. 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 18
  25. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
  26. 2025COSMIC: Generalized Refusal Direction Identification in LLM Activationscited by 15
  27. 2025Understanding and Steering the Cognitive Behaviors of Reasoning Models at Test-Timecited by 10
  28. 2025A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinationscited by 8
  29. 2025Steering Multimodal Large Language Models Decoding for Context-Aware Safetycited by 6
  30. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbationscited by 4
  31. 2025Test-time Diverse Reasoning by Riemannian Activation Steeringcited by 3
  32. 2025On the geometry and topology of representations: the manifolds of modular additioncited by 3
  33. 2026Towards Understanding Steering Strengthcited by 3
  34. 2026Inference-time Alignment via Sparse Junction Steeringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimescited by 2
  2. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 0
  3. 2026Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMscited by 0
  4. 2026Minimally Invasive Steering of Language Modelscited by 0
  5. 2026Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Directioncited by 0
  6. 2026Activation Flow: Manufacturing Activations for Steeringcited by 0
  7. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0