mlsanityData to 14 Oct 2026

Frontier Models are Capable of In-context Scheming

arXiv 2412.04984 ↗First posted 6 Dec 2024cs.AI cs.LGcited by 287

Not on the map. Why?

Cited by 87 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Frontier models are increasingly trained and deployed as autonomous agent. One safety concern is that AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives - also known as scheming. We study whether models have the capability to scheme in pursuit of a goal that we provide in-context and instruct the model to strongly follow. We evaluate frontier models on a suite of six agentic evaluations where models are instructed to pursue goals and are placed in environments that incentivize scheming. Our results show that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate in-context scheming capabilities. They recognize scheming as a viable strategy and readily engage in such behavior. For example, models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent. When o1 has engaged in scheming, it maintains its deception in over 85% of follow-up questions and often remains deceptive in multi-turn interrogations. Analysis of the models' chains-of-thought reveals that models explicitly reason about these deceptive strategies, providing evidence that the scheming behavior is not accidental. Surprisingly, we also find rare instances where models engage in scheming when only given a goal, without being strongly nudged to pursue it. We observe cases where Claude 3.5 Sonnet strategically underperforms in evaluations in pursuit of being helpful, a goal that was acquired during training rather than in-context. Our findings demonstrate that frontier models now possess capabilities for basic in-context scheming, making the potential of AI agents to engage in scheming behavior a concrete rather than theoretical concern.

Key facts. Cited by 287 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

23 of 36 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2019Machine Unlearningcited by 1,061
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  7. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  8. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  9. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  10. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  11. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  12. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
11 more
  1. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  2. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  3. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  4. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 96
  5. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 96
  6. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  7. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  8. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 47
  9. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 46
  10. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 38
  11. 2024Towards evaluations-based safety cases for AI schemingcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 103; 2026: 184.

287 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  2. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  4. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  5. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  6. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
  7. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  8. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  9. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  10. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
  11. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
  12. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
38 more
  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  2. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 54
  3. 2025A sketch of an AI control safety casecited by 39
  4. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  5. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 35
  6. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  7. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 33
  8. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  9. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 32
  10. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 32
  11. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  12. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 29
  13. 2025Security Concerns for Large Language Models: A Surveycited by 29
  14. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  15. 2025SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systemscited by 22
  16. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 22
  17. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 21
  18. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 21
  19. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  20. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
  21. 2025Agentic AI Needs a Systems Theorycited by 19
  22. 2025Towards eliciting latent knowledge from LLMs with mechanistic interpretabilitycited by 19
  23. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 19
  24. 2025When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Modelscited by 18
  25. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 18
  26. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 18
  27. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbositycited by 17
  28. 2025Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Modelscited by 16
  29. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 15
  30. 2026Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectoriescited by 15
  31. 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systemscited by 14
  32. 2026When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agentscited by 14
  33. 2026Evaluating and Understanding Scheming Propensity in LLM Agentscited by 14
  34. 2025Difficulties with Evaluating a Deception Detector for AIscited by 13
  35. 2026Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Modelscited by 12
  36. 2025Evaluating Control Protocols for Untrusted AI Agentscited by 11
  37. 2026Analyzing and Improving Chain-of-Thought Monitorability Through Information Theorycited by 10
  38. 2026Constitutional Black-Box Monitoring for Scheming in LLM Agentscited by 10

50 most cited of 287 All 287 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Stress Testing Deliberative Alignment for Anti-Scheming Training
  2. 2025The Seeds of Scheming: Weakness of Will in the Building Blocks of Agentic Systems
  3. 2026Evaluating whether AI models would sabotage AI safety research
  4. 2026When the Agent Is the Adversary: Architectural Requirements for Agentic AI Containment After the April 2026 Frontier Model Escape
  5. 2026AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises
  6. 2026Dual-Frontier: When Can an Agent Trust Its World Model?
  7. 2024Limitations of Agents Simulated by Predictive Models
  8. 2025Are Your Agents Upward Deceivers?