# Taking control: Policies to address extinction risks from advanced AI

arXiv: https://arxiv.org/abs/2310.20563 · HTML: https://mlsanity.com/paper/2310.20563/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 18 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 2 of 2)

- [What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring](https://mlsanity.com/paper/2303.11341/) (2023)
- [Nuclear Arms Control Verification and Lessons for AI Treaties](https://mlsanity.com/paper/2304.04123/) (2023)

## Built on by (most cited first, 3 of 3)

- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty](https://mlsanity.com/paper/2503.18956/) (2025)
- [Combatting deepfakes: Policies to address national security threats and rights violations](https://mlsanity.com/paper/2402.09581/) (2024)

## Abstract (arXiv)

This paper provides policy recommendations to reduce extinction risks from advanced artificial intelligence (AI). First, we briefly provide background information about extinction risks from AI. Second, we argue that voluntary commitments from AI companies would be an inappropriate and insufficient response. Third, we describe three policy proposals that would meaningfully address the threats from advanced AI: (1) establishing a Multinational AGI Consortium to enable democratic oversight of advanced AI (MAGIC), (2) implementing a global cap on the amount of computing power used to train an AI system (global compute cap), and (3) requiring affirmative safety evaluations to ensure that risks are kept below acceptable levels (gating critical experiments). MAGIC would be a secure, safety-focused, internationally-governed institution responsible for reducing risks from advanced AI and performing research to safely harness the benefits of AI. MAGIC would also maintain emergency response infrastructure (kill switch) to swiftly halt AI development or withdraw model deployment in the event of an AI-related emergency. The global compute cap would end the corporate race toward dangerous AI systems while enabling the vast majority of AI innovation to continue unimpeded. Gating critical experiments would ensure that companies developing powerful AI systems are required to present affirmative evidence that these models keep extinction risks below an acceptable threshold. After describing these recommendations, we propose intermediate steps that the international community could take to implement these proposals and lay the groundwork for international coordination around advanced AI.