mlsanityData to 14 Oct 2026

Power-seeking can be probable and predictive for trained agents

arXiv 2304.06528 ↗First posted 13 Apr 2023cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Power-seeking behavior is a key source of risk from advanced AI, but our theoretical understanding of this phenomenon is relatively limited. Building on existing theoretical results demonstrating power-seeking incentives for most reward functions, we investigate how the training process affects power-seeking incentives and show that they are still likely to hold for trained agents under some simplifying assumptions. We formally define the training-compatible goal set (the set of goals consistent with the training rewards) and assume that the trained agent learns a goal from this set. In a setting where the trained agent faces a choice to shut down or avoid shutdown in a new situation, we prove that the agent is likely to avoid shutdown. Thus, we show that power-seeking incentives can be probable (likely to arise for trained agents) and predictive (allowing us to predict undesirable behavior in new situations).

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 6 of its 9 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

6 of 9 references.

References · 6

  1. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  2. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  3. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  4. 2019Optimal Policies Tend to Seek Powercited by 119
  5. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  6. 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 5; 2026: 11.

25 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023Large Language Model Alignment: A Surveycited by 243
  3. 2023Model evaluation for extreme riskscited by 177
  4. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  5. 2024Large Language Model Safety: A Holistic Surveycited by 59
  6. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  7. 2025An alignment safety case sketch based on debatecited by 16
  8. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 8
  9. 2025Incident Analysis for AI Agentscited by 5
  10. 2025Will artificial agents pursue power by default?cited by 4
  11. 2024CRiskEval: A Chinese Multi-Level Risk Evaluation Benchmark Dataset for Large Language Modelscited by 3
  12. 2026Value-Conflict Diagnostics Reveal Widespread Alignment Faking in Language Modelscited by 3
13 more
  1. 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviorscited by 3
  2. 2024Quantifying stability of non-power-seeking in artificial agentscited by 2
  3. 2025Detecting Malicious AI Agents Through Simulated Interactionscited by 2
  4. 2025Contemplative Artificial Intelligencecited by 1
  5. 2026Steerability of Instrumental-Convergence Tendencies in LLMscited by 1
  6. 2026Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMscited by 1
  7. 2026Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Responsecited by 1
  8. 2026Revisiting the shutdown problemcited by 0
  9. 2026Instrumental convergence and power-seekingcited by 0
  10. 2026Cognitive Digital Twins: Ethical Risks and Governance for AI Systems That Model the Mindcited by 0
  11. 2026Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenouscited by 0
  12. 2026Xeno-Interpretability: Investigating the Alien Minds of LLMscited by 0
  13. 2026Shutdown Sabotage Propensities in Multi-Agent Systemscited by 0