# Parametrically Retargetable Decision-Makers Tend To Seek Power

arXiv: https://arxiv.org/abs/2206.13477 · HTML: https://mlsanity.com/paper/2206.13477/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 36 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 5 of 5)

- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)

## Built on by (most cited first, 19 of 19)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety](https://mlsanity.com/paper/2403.12316/) (2024)
- [A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seeking](https://mlsanity.com/paper/2310.18244/) (2023)
- [Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety](https://mlsanity.com/paper/2305.19223/) (2023)
- [Will artificial agents pursue power by default?](https://mlsanity.com/paper/2506.06352/) (2025)
- [Estimating the Empowerment of Language Model Agents](https://mlsanity.com/paper/2509.22504/) (2025)
- [Towards Shutdownable Agents via Stochastic Choice](https://mlsanity.com/paper/2407.00805/) (2024)
- [When Assisting One Disempowers Another](https://mlsanity.com/paper/2511.04177/) (2025)
- [Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs](https://mlsanity.com/paper/2604.17502/) (2026)
- [AI Systems of Concern](https://mlsanity.com/paper/2310.05876/) (2023)
- [Status Hierarchies in Language Models](https://mlsanity.com/paper/2601.17577/) (2026)
- [The Agent Use of Agent Beings: Agent Cybernetics Is the Missing Science of Foundation Agents](https://mlsanity.com/paper/2605.10754/) (2026)
- [Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous](https://mlsanity.com/paper/2608.14795/) (2026)
- [AI Alignment through a Game-theoretic Lens: A Survey](https://mlsanity.com/paper/2608.27910/) (2026)

## Abstract (arXiv)

If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal. We consider a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable, and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad. We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma's Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.