mlsanityData to 7 Oct 2026

AI Research Considerations for Human Existential Safety (ARCHES)

arXiv 2006.04948 ↗First posted 30 May 2020cs.CY cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Framed in positive terms, this report examines how technical AI research might be steered in a manner that is more attentive to humanity's long-term prospects for survival as a species. In negative terms, we ask what existential risks humanity might face from AI development in the next century, and by what principles contemporary technical research might be directed to address those risks. A key property of hypothetical AI technologies is introduced, called prepotence, which is useful for delineating a variety of potential existential risks from artificial intelligence, even as AI paradigms might shift. A set of auxrefdirtot contemporary research directions are then examined for their potential benefit to existential safety. Each research direction is explained with a scenario-driven motivation, and examples of existing work from which to build. The research directions present their own risks and benefits to society that could occur at various scales of impact, and in particular are not guaranteed to benefit existential safety if major developments in them are deployed without adequate forethought and oversight. As such, each direction is accompanied by a consideration of potentially negative side effects.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 5; 2024: 6; 2025: 8; 2026: 7.

33 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2023Large Language Model Alignment: A Surveycited by 241
  4. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  5. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  6. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  7. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  8. 2022Structured access: an emerging paradigm for safe AI deploymentcited by 40
  9. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  10. 2022Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysiscited by 34
  11. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  12. 2022Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governancecited by 28
21 more
  1. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  2. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  3. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  4. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  5. 2024Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignmentcited by 8
  6. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 7
  7. 2025Spore in the Wild: A Case Study of Spore.fun as an Open-Environment Evolution Experiment with Sovereign AI Agents on TEE-Secured Blockchainscited by 5
  8. 2022Negative Human Rights as a Basis for Long-term AI Safety and Regulationcited by 3
  9. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  10. 2025The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomycited by 3
  11. 2025Supervision policies can shape long-term risk management in general-purpose AI modelscited by 2
  12. 2025Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Marketscited by 2
  13. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  14. 2026Eigenism: Ethics for a Human-AI Futurecited by 1
  15. 2026A dataset of rated conceptual argumentscited by 1
  16. 2026Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populationscited by 1
  17. 2025Servant, Stalker, Predator: How An Honest, Helpful, And Harmless (3H) Agent Unlocks Adversarial Skillscited by 0
  18. 2026Measuring and Eliminating Refusals in Military Large Language Modelscited by 0
  19. 2026Implementing surrogate goals for safer bargaining in LLM-based agentscited by 0
  20. 2026Civilizational Metamaterials: Engineering Coordination Under Capability Gradients and Structural Turbulencecited by 0
  21. 2026Information Limits and Attractor Dynamics in Economies of Frontier LLM Agents: A Pre-Registered Testcited by 0