# Steerability of Instrumental-Convergence Tendencies in LLMs

arXiv: https://arxiv.org/abs/2601.01584 · HTML: https://mlsanity.com/paper/2601.01584/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 48 references that are in the map.
- First posted on arXiv 4 Jan 2026.

## Builds on (most cited first, 30 of 44)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Prompt Injection Attack to Tool Selection in LLM Agents](https://mlsanity.com/paper/2504.19793/) (2025)
- [Tamper-Resistant Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2408.00761/) (2024)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [What Makes and Breaks Safety Fine-tuning? A Mechanistic Study](https://mlsanity.com/paper/2407.10264/) (2024)
- [On Large Language Model Continual Unlearning](https://mlsanity.com/paper/2407.10223/) (2024)
- [On Evaluating the Durability of Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2412.07097/) (2024)
- [WebInject: Prompt Injection Attack to Web Agents](https://mlsanity.com/paper/2505.11717/) (2025)
- [Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs](https://mlsanity.com/paper/2508.06601/) (2025)
- [Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering](https://mlsanity.com/paper/2410.16314/) (2024)
- [Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities](https://mlsanity.com/paper/2502.05209/) (2025)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [Erasing Conceptual Knowledge from Language Models](https://mlsanity.com/paper/2410.02760/) (2024)
- [A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models](https://mlsanity.com/paper/2503.01854/) (2025)
- [Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?](https://mlsanity.com/paper/2502.12206/) (2025)
- [Toward universal steering and monitoring of AI models](https://mlsanity.com/paper/2502.03708/) (2025)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [ReLearn: Unlearning via Learning for Large Language Models](https://mlsanity.com/paper/2502.11190/) (2025)
- [Estimating Worst-Case Frontier Risks of Open-Weight LLMs](https://mlsanity.com/paper/2508.03153/) (2025)
- [Precise In-Parameter Concept Erasure in Large Language Models](https://mlsanity.com/paper/2505.22586/) (2025)
- [CipherPrune: Efficient and Scalable Private Transformer Inference](https://mlsanity.com/paper/2502.16782/) (2025)
- [Universal Jailbreak Suffixes Are Strong Attention Hijackers](https://mlsanity.com/paper/2506.12880/) (2025)
- [Erased but Not Forgotten: How Backdoors Compromise Concept Erasure](https://mlsanity.com/paper/2504.21072/) (2025)

## Built on by (most cited first, 1 of 1)

- [When AI Says It Feels](https://mlsanity.com/paper/2606.05734/) (2026)

## Abstract (arXiv)

We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety--security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or adversarial attacks. Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available at github.com/j-hoscilowicz/instrumental_steering.