Will artificial agents pursue power by default?
Not on the map. Why?
Abstract (arXiv)
Researchers worried about catastrophic risks from advanced AI have argued that we should expect sufficiently capable AI agents to pursue power over humanity because power is a convergent instrumental goal, something that is useful for a wide range of final goals. Others have recently expressed skepticism of these claims. This paper aims to formalize the concepts of instrumental convergence and power-seeking in an abstract, decision-theoretic framework, and to assess the claim that power is a convergent instrumental goal. I conclude that this claim contains at least an element of truth, but might turn out to have limited predictive utility, since an agent's options cannot always be ranked in terms of power in the absence of substantive information about the agent's final goals. However, the fact of instrumental convergence is more predictive for agents who have a good shot at attaining absolute or near-absolute power.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 10 of its 32 references that are in the map. First posted on arXiv 2 Jun 2025.
Builds on
10 of 32 references.
References · 10
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2021Convex optimizationcited by 2,479
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2020Artificial Intelligence, Values and Alignmentcited by 443
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
- 2023An Overview of Catastrophic AI Riskscited by 214
- 2019Optimal Policies Tend to Seek Powercited by 115
- 2022Is Power-Seeking AI an Existential Risk?cited by 101
- 2023Power-seeking can be probable and predictive for trained agentscited by 23
- 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviorscited by 2
- 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 0
- 2026Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenouscited by 0
- 2026Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous14 Aug 2026
- 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors7 May 2026
- 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agents24 Jan 2026