mlsanityData to 7 Oct 2026

Evaluating and Understanding Scheming Propensity in LLM Agents

arXiv 2603.01608 ↗First posted 2 Mar 2026cs.AIcited by 11

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As frontier language models are increasingly deployed as autonomous agents pursuing complex, long-term objectives, there is increased risk of scheming: agents covertly pursuing misaligned goals. Prior work has focused on showing agents are capable of scheming, but their propensity to scheme in realistic scenarios remains underexplored. To understand when agents scheme, we decompose scheming incentives into agent factors and environmental factors. We develop realistic settings allowing us to systematically vary these factors, each with scheming opportunities for agents that pursue instrumentally convergent goals such as self-preservation, resource acquisition, and goal-guarding. We find only minimal instances of scheming despite high environmental incentives, and show this is unlikely due to evaluation awareness. While inserting adversarially-designed prompt snippets that encourage agency and goal-directedness into an agent's system prompt can induce high scheming rates, snippets used in real agent scaffolds rarely do. Surprisingly, in model organisms (Hubinger et al., 2023) built with these snippets, scheming behavior is remarkably brittle: removing a single tool can drop the scheming rate from 59% to 3%, and increasing oversight can raise rather than deter scheming by up to 25%. Our incentive decomposition enables systematic measurement of scheming propensity in settings relevant for deployment, which is necessary as agents are entrusted with increasingly consequential tasks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 19 of its 49 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

19 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents4 Jun

Other references · 18

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2024Alignment faking in large language modelscited by 316
  3. 2024Frontier Models are Capable of In-context Schemingcited by 252
  4. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  5. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  6. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  7. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  8. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  9. 2025Stress Testing Deliberative Alignment for Anti-Scheming Trainingcited by 71
  10. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 58
  11. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  12. 2024Towards evaluations-based safety cases for AI schemingcited by 32
6 more
  1. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 30
  2. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 19
  3. 2025Lessons from a Chimp: AI "Scheming" and the Quest for Ape Languagecited by 16
  4. 2025Auditing Games for Sandbaggingcited by 10
  5. 2025From surveillance to signalling: escalation channels as environmental controls for agentic AIcited by 6
  6. 2025Scheming Ability in LLM-to-LLM Strategic Interactionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Gram: Assessing sabotage propensities via automated alignment auditingcited by 3
  2. 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviorscited by 2
  3. 2026Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalationcited by 2
  4. 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 1
  5. 2026LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMscited by 1
  6. 2026SysAdmin: Measuring Instrumental Power-Seeking in Frontier AIcited by 1
  7. 2026Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent Systemscited by 1
  8. 2026Realistic honeypot evaluations for scheming propensitycited by 0
  9. 2026AI Researchers Must Help Lead Arms Control to Mitigate Military AI Riskscited by 0
  10. 2026Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Sciencecited by 0
  11. 2026SchemeArena: Factorized Stress Testing of Scheming in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Seeds of Scheming: Weakness of Will in the Building Blocks of Agentic Systems
  2. 2026Constitutional Black-Box Monitoring for Scheming in LLM Agents
  3. 2026A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
  4. 2025Technical Report: Evaluating Goal Drift in Language Model Agents
  5. 2026Large Language Model Agents Are Not Always Faithful Self-Evolvers
  6. 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
  7. 2026Inherited Goal Drift: Contextual Pressure Can Undermine Agentic Goals
  8. 2025What Do LLM Agents Do When Left Alone? Evidence of Spontaneous Meta-Cognitive Patterns