mlsanityData to 14 Oct 2026

Is Power-Seeking AI an Existential Risk?

arXiv 2206.13353 ↗First posted 16 Jun 2022cs.CY cs.AI cs.LGcited by 105

Not on the map. Why?

Abstract (arXiv)

This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire -- especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. Second, I formulate and evaluate a more specific six-premise argument that creating agents of this kind will lead to existential catastrophe by 2070. On this argument, by 2070: (1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe. I assign rough subjective credences to the premises in this argument, and I end up with an overall estimate of ~5% that an existential catastrophe of this kind will occur by 2070. (May 2022 update: since making this report public in April 2021, my estimate here has gone up, and is now at >10%.)

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Builds on 20 of its 108 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

20 of 108 references.

References · 20

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  7. 2018AI safety via debatecited by 400
  8. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  9. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  10. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 209
  11. 2018Categorizing Variants of Goodhart's Lawcited by 149
  12. 2018Supervising strong learners by amplifying weak expertscited by 145
8 more
  1. 2019Optimal Policies Tend to Seek Powercited by 119
  2. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 43
  3. 2017When Will AI Exceed Human Performance? Evidence from AI Expertscited by 41
  4. 2020AI Research Considerations for Human Existential Safety (ARCHES)cited by 33
  5. 2019The Role of Cooperation in Responsible AI Developmentcited by 21
  6. 2020Avoiding Tampering Incentives in Deep RL via Decoupled Approvalcited by 12
  7. 2020REALab: An Embedded Perspective on Tamperingcited by 8
  8. 2017On the Impossibility of Supersized Machinescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 21; 2024: 19; 2025: 27; 2026: 32.

105 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  4. 2023Are aligned neural networks adversarially aligned?cited by 315
  5. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  6. 2023Large Language Model Alignment: A Surveycited by 243
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  8. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  9. 2023Model evaluation for extreme riskscited by 177
  10. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
38 more
  1. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  2. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  3. 2024Large Language Model Safety: A Holistic Surveycited by 59
  4. 2023Unifying Grokking and Double Descentcited by 50
  5. 2024Beyond Preferences in AI Alignmentcited by 49
  6. 2024Thousands of AI Authors on the Future of AIcited by 48
  7. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  8. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  9. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 33
  10. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  11. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  12. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 32
  13. 2023Power-seeking can be probable and predictive for trained agentscited by 25
  14. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 25
  15. 2023Natural Selection Favors AIs over Humanscited by 24
  16. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  17. 2024Two Types of AI Existential Risk: Decisive and Accumulativecited by 22
  18. 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19
  19. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  20. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 16
  21. 2022Current and Near-Term AI as a Potential Existential Risk Factorcited by 14
  22. 2025Distributional AGI Safetycited by 12
  23. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  24. 2024Societal Adaptation to Advanced AIcited by 10
  25. 2025The Steganographic Potentials of Language Modelscited by 10
  26. 2024OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safetycited by 9
  27. 2026AI Assistance Reduces Persistence and Hurts Independent Performancecited by 9
  28. 2023Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standardscited by 8
  29. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 8
  30. 2025Principles for Responsible AI Consciousness Researchcited by 7
  31. 2025Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Modelscited by 7
  32. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  33. 2023ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?cited by 6
  34. 2026SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitorscited by 6
  35. 2025AI Safety vs. AI Security: Demystifying the Distinction and Boundariescited by 5
  36. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 5
  37. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  38. 2025Will artificial agents pursue power by default?cited by 4

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The AI Risk Spectrum: From Dangerous Capabilities to Existential Threats
  2. 2024Taking AI Welfare Seriously
  3. 2023Absolutist AI
  4. 2023Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety
  5. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
  6. 2025Neurodivergent Influenceability as a Contingent Solution to the AI Alignment Problem
  7. 2025Emergent Risk Awareness in Rational Agents under Resource Constraints
  8. 2025Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power