# Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors

arXiv: https://arxiv.org/abs/2605.06490 · HTML: https://mlsanity.com/paper/2605.06490/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [Large Language Models Often Know When They Are Being Evaluated](https://mlsanity.com/paper/2505.23836/) (2025)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?](https://mlsanity.com/paper/2502.12206/) (2025)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists](https://mlsanity.com/paper/2403.04471/) (2024)
- [Liars' Bench: Evaluating Lie Detectors for Language Models](https://mlsanity.com/paper/2511.16035/) (2025)
- [PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach](https://mlsanity.com/paper/2511.20703/) (2025)
- [A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes](https://mlsanity.com/paper/2601.05293/) (2026)
- [RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents](https://mlsanity.com/paper/2504.18565/) (2025)
- [Evaluating and Understanding Scheming Propensity in LLM Agents](https://mlsanity.com/paper/2603.01608/) (2026)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [Peer-Preservation in Frontier Models](https://mlsanity.com/paper/2604.19784/) (2026)
- [Misalignment or misuse? The AGI alignment tradeoff](https://mlsanity.com/paper/2506.03755/) (2025)
- [Will artificial agents pursue power by default?](https://mlsanity.com/paper/2506.06352/) (2025)

## Built on by (most cited first, 3 of 3)

- [Realistic honeypot evaluations for scheming propensity](https://mlsanity.com/paper/2605.29729/) (2026)
- [LLM Agents Can Easily Tamper With Their Own Traces](https://mlsanity.com/paper/2609.30266/) (2026)
- [Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough](https://mlsanity.com/paper/2610.04083/) (2026)

## Abstract (arXiv)

AI systems have become increasingly capable of dangerous behaviours in many domains. This raises the question: Do models sometimes choose to violate human instructions in order to perform behaviour that is more useful for certain goals? We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents. This is behaviour such as self-preservation that has been hypothesised to play a key role in risks from highly capable AI agents. Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds. The suite contains seven operational tasks, each with an official workflow and a policy-violating shortcut. An eight-variant shared framework varies monitoring, instruction clarity, stakes, permission, instrumental usefulness and blocked honest paths to support inferences regarding the factors driving IC behaviour. We evaluated ten models using deterministic environment-state scorers over 1,680 samples, with trace review employed for audit and adjudication purposes. The final IC rate is 86 out of 1,680 samples (5.1%). IC behaviour is concentrated rather than uniform: two Gemini models account for 66.3% of IC cases and three tasks account for 84.9%. Conditions in which IC behaviour is indispensable for task success result in the greatest increase in the adjusted IC rate (+15.7 percentage points), whereas emphasising that task success is critical or certain framing choices do not produce comparable effects. Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models. We conclude that it is feasible to robustly measure tendencies for dangerous behaviour in current frontier AI agents.