mlsanityData to 14 Oct 2026

Scheming AIs: Will AIs fake alignment during training in order to get power?

arXiv 2311.08379 ↗First posted 14 Nov 2023cs.CY cs.AI cs.LGcited by 97

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This report examines whether advanced AIs that perform well in training will be doing so in order to gain power later -- a behavior I call "scheming" (also sometimes called "deceptive alignment"). I conclude that scheming is a disturbingly plausible outcome of using baseline machine learning methods to train goal-directed AIs sophisticated enough to scheme (my subjective probability on such an outcome, given these conditions, is roughly 25%). In particular: if performing well in training is a good strategy for gaining power (as I think it might well be), then a very wide variety of goals would motivate scheming -- and hence, good training performance. This makes it plausible that training might either land on such a goal naturally and then reinforce it, or actively push a model's motivations towards such a goal as an easy way of improving performance. What's more, because schemers pretend to be aligned on tests designed to reveal their motivations, it may be quite difficult to tell whether this has occurred. However, I also think there are reasons for comfort. In particular: scheming may not actually be such a good strategy for gaining power; various selection pressures in training might work against schemer-like goals (for example, relative to non-schemers, schemers need to engage in extra instrumental reasoning, which might harm their training performance); and we may be able to increase such pressures intentionally. The report discusses these and a wide variety of other considerations in detail, and it suggests an array of empirical research directions for probing the topic further.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Nov 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 31; 2026: 49.

97 papers.

Sort:

By area:

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  2. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  3. 2024Alignment faking in large language modelscited by 360
  4. 2024Frontier Models are Capable of In-context Schemingcited by 287
  5. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  6. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  7. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  8. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  9. 2024Universal Neurons in GPT2 Language Modelscited by 102
  10. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  11. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 88
  12. 2024Sabotage Evaluations for Frontier Modelscited by 52
38 more
  1. 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 44
  2. 2025A sketch of an AI control safety casecited by 39
  3. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 38
  4. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  5. 2025Safety Guardrails for LLM-Enabled Robotscited by 35
  6. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 35
  7. 2025Towards Understanding Subliminal Learning: When and How Hidden Biases Transfercited by 34
  8. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  9. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 31
  10. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  11. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 24
  12. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 22
  13. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  14. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 19
  15. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  16. 2026Evaluating and Understanding Scheming Propensity in LLM Agentscited by 14
  17. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13
  18. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 12
  19. 2025Evaluating Control Protocols for Untrusted AI Agentscited by 11
  20. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 11
  21. 2026Constitutional Black-Box Monitoring for Scheming in LLM Agentscited by 10
  22. 2026LinuxArena: A Control Setting for AI Agents in Live Production Software Environmentscited by 8
  23. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  24. 2025Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfarecited by 7
  25. 2026Propensity Inference: Environmental Contributors to LLM Behaviourcited by 7
  26. 2025Beyond Black-Box Obfuscation: Mechanistic Analysis and Defense of White-Box Monitorscited by 6
  27. 2025Optimizing AI Agent Attacks With Synthetic Datacited by 6
  28. 2026Training Agents to Self-Report Misbehaviorcited by 6
  29. 2025Scheming Ability in LLM-to-LLM Strategic Interactionscited by 5
  30. 2026Emergent Risks in Generative Multi-Agent Systemscited by 5
  31. 2026Distributed Attacks in Persistent-State AI Controlcited by 5
  32. 2025Empirical Evidence for Alignment Faking in a Small LLM and Prompt-Based Mitigation Techniquescited by 4
  33. 2026Removing Sandbagging in LLMs by Training with Weak Supervisioncited by 4
  34. 2026LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awarenesscited by 4
  35. 2026Training Deliberative Monitors for Black-Box Scheming Detectioncited by 4
  36. 2026Multi-Agent Systems Should be Treated as Principal-Agent Problemscited by 3
  37. 2026Basic Legibility Protocols Improve Trusted Monitoringcited by 3
  38. 2026Value-Conflict Diagnostics Reveal Widespread Alignment Faking in Language Modelscited by 3

50 most cited of 97 All 97 (JSON)