mlsanityData to 14 Oct 2026

Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors

arXiv 2605.06490 ↗First posted 7 May 2026cs.AI cs.CYcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

AI systems have become increasingly capable of dangerous behaviours in many domains. This raises the question: Do models sometimes choose to violate human instructions in order to perform behaviour that is more useful for certain goals? We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents. This is behaviour such as self-preservation that has been hypothesised to play a key role in risks from highly capable AI agents. Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds. The suite contains seven operational tasks, each with an official workflow and a policy-violating shortcut. An eight-variant shared framework varies monitoring, instruction clarity, stakes, permission, instrumental usefulness and blocked honest paths to support inferences regarding the factors driving IC behaviour. We evaluated ten models using deterministic environment-state scorers over 1,680 samples, with trace review employed for audit and adjudication purposes. The final IC rate is 86 out of 1,680 samples (5.1%). IC behaviour is concentrated rather than uniform: two Gemini models account for 66.3% of IC cases and three tasks account for 84.9%. Conditions in which IC behaviour is indispensable for task success result in the greatest increase in the adjusted IC rate (+15.7 percentage points), whereas emphasising that task success is critical or certain framing choices do not produce comparable effects. Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models. We conclude that it is feasible to robustly measure tendencies for dangerous behaviour in current frontier AI agents.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 34 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

21 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Evaluating and Understanding Scheming Propensity in LLM Agents2 Mar
  2. 2025RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents21 Apr

Other references · 19

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2024Alignment faking in large language modelscited by 360
  4. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  5. 2024Frontier Models are Capable of In-context Schemingcited by 287
  6. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
  7. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  8. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  9. 2019Optimal Policies Tend to Seek Powercited by 119
  10. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 33
  11. 2023Power-seeking can be probable and predictive for trained agentscited by 25
  12. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
7 more
  1. 2025Liars' Bench: Evaluating Lie Detectors for Language Modelscited by 22
  2. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 22
  3. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypescited by 22
  4. 2025Difficulties with Evaluating a Deception Detector for AIscited by 13
  5. 2026Peer-Preservation in Frontier Modelscited by 9
  6. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  7. 2025Will artificial agents pursue power by default?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Realistic honeypot evaluations for scheming propensitycited by 1
  2. 2026LLM Agents Can Easily Tamper With Their Own Tracescited by 0
  3. 2026Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enoughcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Set-shifting Behavioral Test for Harnessed Agents
  2. 2026Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure
  3. 2026On the Behavioral Traits of LLM Agents
  4. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agents
  5. 2025Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components
  6. 2025Behavioral Inference at Scale: The Fundamental Asymmetry Between Motivations and Belief Systems
  7. 2026Grounded Scaling: Why Agentic AI Needs Deterministic Environments
  8. 2026Gram: Assessing sabotage propensities via automated alignment auditing