mlsanityData to 14 Oct 2026

MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptation

arXiv 2609.38718 ↗First posted 30 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Steering large language models typically relies on linear, context-independent interventions in activation space, an assumption that recent work has challenged and that can induce an information bottleneck when a fixed representation must encode many behavioral distinctions. We introduce MetaSteer, a method that learns nonlinear interventions with context-dependent effects and applies them to attention projection matrices, producing activation effects that vary with the input context by construction and requiring no linear concept-geometry assumption. Framed as preference-based optimization, MetaSteer is trained once on a pooled preference corpus and transferred zero-shot to unseen concepts and out-of-distribution contexts. We find that, despite using low-rank adapters, MetaSteer induces structured, context-dependent changes in hidden-state trajectories while partially preserving aspects of their local trajectory dynamics, including velocity and curvature. We evaluate MetaSteer on three controlled text-generation benchmarks and three agentic settings across multiple model families and scales. MetaSteer matches or outperforms strong task-specific steering baselines on most aggregate comparisons in the zero-shot regime. Across the evaluated settings, stronger text-generation steering is associated with stronger agentic steering performance. We further discuss geometric trajectory effects, capability retention, and safety considerations raised by transferable steering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 64 of its 67 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

64 of 67 references.

References · 64

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2023Steering Language Models With Activation Engineeringcited by 922
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
38 more
  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  3. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  4. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  5. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  6. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 223
  7. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  8. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  9. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  10. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
  11. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  12. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  13. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 92
  14. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
  15. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  16. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 62
  17. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  18. 2025The Origins of Representation Manifolds in Large Language Modelscited by 51
  19. 2024ICLR: In-Context Learning of Representationscited by 50
  20. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 46
  21. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 38
  22. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 35
  23. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Modelscited by 34
  24. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Modelscited by 29
  25. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 27
  26. 2025HyperSteer: Activation Steering at Scale with Hypernetworkscited by 23
  27. 2026Sycophancy Hides Linearly in the Attention Headscited by 23
  28. 2025Improved Representation Steering for Language Modelscited by 21
  29. 2026ODESteer: A Unified ODE-Based Steering Framework for LLM Alignmentcited by 19
  30. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representationscited by 17
  31. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 17
  32. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 14
  33. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 14
  34. 2025SocialEval: Evaluating Social Intelligence of Large Language Modelscited by 12
  35. 2025Improving LLM Reasoning through Interpretable Role-Playing Steeringcited by 12
  36. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Featurescited by 12
  37. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmarkcited by 12
  38. 2024Neuron-based Personality Trait Induction in Large Language Modelscited by 9

50 most cited of 64 All 64 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention
  2. 2026IDEEA: training-free Input-Dependent stEEring via Activation cluster matching
  3. 2026Disentangling Steering Vectors
  4. 2026Activation Steering via Generative Causal Mediation
  5. 2025Understanding (Un)Reliability of Steering Vectors in Language Models
  6. 2026UniSteer: Text-Guided Flow Matching in Activation Space for Versatile LLM Steering
  7. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics
  8. 2026Minimally Invasive Steering of Language Models