# Optimal Policies Tend to Seek Power

arXiv: https://arxiv.org/abs/1912.01683 · HTML: https://mlsanity.com/paper/1912.01683/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 30 of 119)

- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation](https://mlsanity.com/paper/2306.11706/) (2023)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Preventing Language Models From Hiding Their Reasoning](https://mlsanity.com/paper/2310.18512/) (2023)
- [X-Risk Analysis for AI Research](https://mlsanity.com/paper/2206.05862/) (2022)
- [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Managing extreme AI risks amid rapid progress](https://mlsanity.com/paper/2310.17688/) (2023)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [Improving Generalization of Alignment with Human Preferences through Group Invariant Learning](https://mlsanity.com/paper/2310.11971/) (2023)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [REvolve: Reward Evolution with Large Language Models using Human Feedback](https://mlsanity.com/paper/2406.01309/) (2024)
- [The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists](https://mlsanity.com/paper/2403.04471/) (2024)
- [Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs](https://mlsanity.com/paper/2601.11369/) (2026)
- [Parametrically Retargetable Decision-Makers Tend To Seek Power](https://mlsanity.com/paper/2206.13477/) (2022)

## Abstract (arXiv)

Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.