# Against racing to AGI: Cooperation, deterrence, and catastrophic risks

arXiv: https://arxiv.org/abs/2507.21839 · HTML: https://mlsanity.com/paper/2507.21839/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 88 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 30 of 42)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Foundational Challenges in Assuring Alignment and Safety of Large Language Models](https://mlsanity.com/paper/2404.09932/) (2024)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [Escalation Risks from Language Models in Military and Diplomatic Decision-Making](https://mlsanity.com/paper/2401.03408/) (2024)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Thousands of AI Authors on the Future of AI](https://mlsanity.com/paper/2401.02843/) (2024)
- [Safety Cases: How to Justify the Safety of Advanced AI Systems](https://mlsanity.com/paper/2403.10462/) (2024)
- [Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development](https://mlsanity.com/paper/2501.16946/) (2025)
- [An Approach to Technical AGI Safety and Security](https://mlsanity.com/paper/2504.01849/) (2025)
- [Ctrl-Z: Controlling AI Agents via Resampling](https://mlsanity.com/paper/2504.10374/) (2025)
- [Infrastructure for AI Agents](https://mlsanity.com/paper/2501.10114/) (2025)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI](https://mlsanity.com/paper/2306.06924/) (2023)
- [Two Types of AI Existential Risk: Decisive and Accumulative](https://mlsanity.com/paper/2401.07836/) (2024)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [Safety cases for frontier AI](https://mlsanity.com/paper/2410.21572/) (2024)

## Built on by (most cited first, 2 of 2)

- [AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?](https://mlsanity.com/paper/2510.11235/) (2025)
- [Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence](https://mlsanity.com/paper/2605.01297/) (2026)

## Abstract (arXiv)

AGI Racing is the view that it is in the self-interest of major actors in AI development, especially powerful nations, to accelerate their frontier AI development to build highly capable AI, especially artificial general intelligence (AGI), before competitors have a chance. We argue against AGI Racing. First, the downsides of racing to AGI are much higher than portrayed by this view. Racing to AGI would substantially increase catastrophic risks from AI, including nuclear instability, and undermine the prospects of technical AI safety research to be effective. Second, the expected benefits of racing may be lower than proponents of AGI Racing hold. In particular, it is questionable whether winning the race enables complete domination over losers. Third, international cooperation and coordination, and perhaps carefully crafted deterrence measures, constitute viable alternatives to racing to AGI which have much smaller risks and promise to deliver most of the benefits that racing to AGI is supposed to provide. Hence, racing to AGI is not in anyone's self-interest as other actions, particularly incentivizing and seeking international cooperation around AI issues, are preferable.