mlsanityData to 7 Oct 2026

Optimal Policies Tend to Seek Power

arXiv 1912.01683 ↗First posted 3 Dec 2019cs.AIcited by 115

Not on the map. Why?

Abstract (arXiv)

Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 19; 2024: 21; 2025: 24; 2026: 38.

115 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2021Unsolved Problems in ML Safetycited by 291
  5. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2022Defining and Characterizing Reward Hackingcited by 184
  8. 2023Model evaluation for extreme riskscited by 167
  9. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  10. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  11. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  12. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
38 more
  1. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  2. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  3. 2023Eight Things to Know about Large Language Modelscited by 73
  4. 2023Preventing Language Models From Hiding Their Reasoningcited by 62
  5. 2022X-Risk Analysis for AI Researchcited by 60
  6. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  7. 2023Unifying Grokking and Double Descentcited by 49
  8. 2023Managing extreme AI risks amid rapid progresscited by 48
  9. 2024Beyond Preferences in AI Alignmentcited by 46
  10. 2024Towards evaluations-based safety cases for AI schemingcited by 32
  11. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  12. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  13. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  14. 2023Power-seeking can be probable and predictive for trained agentscited by 23
  15. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  16. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 22
  17. 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19
  18. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  19. 2024Contestable AI needs Computational Argumentationcited by 12
  20. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  21. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  22. 2025Plasticity as the Mirror of Empowermentcited by 11
  23. 2023Human Control: Definitions and Algorithmscited by 10
  24. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistantscited by 10
  25. 2023Incentivizing honest performative predictions with proper scoring rulescited by 9
  26. 2024OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safetycited by 9
  27. 2024On Goodhart's law, with an application to value alignmentcited by 9
  28. 2025Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quiltcited by 9
  29. 2021Learning Altruistic Behaviours in Reinforcement Learning without External Rewardscited by 8
  30. 2023Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standardscited by 8
  31. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  32. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 7
  33. 2025AI Testing Should Account for Sophisticated Strategic Behaviourcited by 6
  34. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 6
  35. 2024Active Inference and Human--Computer Interactioncited by 4
  36. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  37. 2022Formalizing the Problem of Side Effect Regularizationcited by 3
  38. 2023Measuring Value Alignmentcited by 3

50 most cited of 115 All 115 (JSON)