mlsanityData to 14 Oct 2026

Steerability of Instrumental-Convergence Tendencies in LLMs

arXiv 2601.01584 ↗First posted 4 Jan 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety--security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or adversarial attacks. Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available at github.com/j-hoscilowicz/instrumental_steering.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 48 references that are in the map. First posted on arXiv 4 Jan 2026.

Builds on

44 of 48 references.

References · 44

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  4. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  5. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  6. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  7. 2019Optimal Policies Tend to Seek Powercited by 119
  8. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  9. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  10. 2025Safety in Large Reasoning Models: A Surveycited by 64
  11. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  12. 2024On Large Language Model Continual Unlearningcited by 52
32 more
  1. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  2. 2025WebInject: Prompt Injection Attack to Web Agentscited by 51
  3. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 42
  4. 2024Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineeringcited by 39
  5. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  6. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 35
  7. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  8. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 34
  9. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 33
  10. 2025Toward universal steering and monitoring of AI modelscited by 30
  11. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 27
  12. 2023Power-seeking can be probable and predictive for trained agentscited by 25
  13. 2025ReLearn: Unlearning via Learning for Large Language Modelscited by 22
  14. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 19
  15. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  16. 2025CipherPrune: Efficient and Scalable Private Transformer Inferencecited by 10
  17. 2025Universal Jailbreak Suffixes Are Strong Attention Hijackerscited by 10
  18. 2025Erased but Not Forgotten: How Backdoors Compromise Concept Erasurecited by 9
  19. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  20. 2023Evaluating Shutdown Avoidance of Language Models in Textual Scenarioscited by 8
  21. 2025STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Modelscited by 8
  22. 2025AI Awarenesscited by 7
  23. 2025AI Deception: Risks, Dynamics, and Controlscited by 7
  24. 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translationcited by 6
  25. 2025Jailbreak Transferability Emerges from Shared Representationscited by 6
  26. 2025Security Steerability is All You Needcited by 4
  27. 2025Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projectioncited by 4
  28. 2024A Survey on Private Transformer Inferencecited by 3
  29. 2025Adversarial Confusion Attack: Disrupting Multimodal Large Language Modelscited by 3
  30. 2025The Economics of p(doom): Scenarios of Existential Risk and Economic Growth in the Age of Transformative AIcited by 2
  31. 2025Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descentcited by 2
  32. 2025Manipulating Transformer-Based Models: Controllability, Steerability, and Robust Interventionscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When AI Says It Feelscited by 0