# A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seeking

arXiv: https://arxiv.org/abs/2310.18244 · HTML: https://mlsanity.com/paper/2310.18244/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [The Problem with Metrics is a Fundamental Problem for AI](https://mlsanity.com/paper/2002.08512/) (2020)
- [AI Research Considerations for Human Existential Safety (ARCHES)](https://mlsanity.com/paper/2006.04948/) (2020)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [Over-Optimization of Academic Publishing Metrics: Observing Goodhart's Law in Action](https://mlsanity.com/paper/1809.07841/) (2018)
- [Parametrically Retargetable Decision-Makers Tend To Seek Power](https://mlsanity.com/paper/2206.13477/) (2022)
- [Growth, degrowth, and the challenge of artificial superintelligence](https://mlsanity.com/paper/1905.04288/) (2019)
- [Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence](https://mlsanity.com/paper/1810.10862/) (2018)
- [An Unethical Optimization Principle](https://mlsanity.com/paper/1911.05116/) (2019)

## Built on by (most cited first, 8 of 8)

- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process](https://mlsanity.com/paper/2506.01080/) (2025)
- [Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communities](https://mlsanity.com/paper/2604.05339/) (2026)
- [CRiskEval: A Chinese Multi-Level Risk Evaluation Benchmark Dataset for Large Language Models](https://mlsanity.com/paper/2406.04752/) (2024)
- [Mitigating loss of control in advanced AI systems through instrumental goal trajectories](https://mlsanity.com/paper/2602.01699/) (2026)
- [An Aristotelian ontology of instrumental goals: Structural features to be managed and not failures to be eliminated](https://mlsanity.com/paper/2510.25471/) (2025)
- [Agentic AI, Retrieval-Augmented Generation, and the Institutional Turn: Legal Architectures and Financial Governance in the Age of Distributional AGI](https://mlsanity.com/paper/2603.13244/) (2026)
- [A three-dimensional typology of agency for advanced AI systems](https://mlsanity.com/paper/2608.20041/) (2026)

## Abstract (arXiv)

Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose existential risks. This paper reviews the evidence for existential risks from AI via misalignment, where AI systems develop goals misaligned with human values, and power-seeking, where misaligned AIs actively seek power. The review examines empirical findings, conceptual arguments and expert opinion relating to specification gaming, goal misgeneralization, and power-seeking. The current state of the evidence is found to be concerning but inconclusive regarding the existence of extreme forms of misaligned power-seeking. Strong empirical evidence of specification gaming combined with strong conceptual evidence for power-seeking make it difficult to dismiss the possibility of existential risk from misaligned power-seeking. On the other hand, to date there are no public empirical examples of misaligned power-seeking in AI systems, and so arguments that future systems will pose an existential risk remain somewhat speculative. Given the current state of the evidence, it is hard to be extremely confident either that misaligned power-seeking poses a large existential risk, or that it poses no existential risk. The fact that we cannot confidently rule out existential risk from AI via misaligned power-seeking is cause for serious concern.