# AGI Safety Literature Review

arXiv: https://arxiv.org/abs/1805.01109 · HTML: https://mlsanity.com/paper/1805.01109/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2018.

## Built on by (most cited first, 28 of 28)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery](https://mlsanity.com/paper/2310.18356/) (2023)
- [A Survey on AI Assurance](https://mlsanity.com/paper/2111.07505/) (2021)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Current and Near-Term AI as a Potential Existential Risk Factor](https://mlsanity.com/paper/2209.10604/) (2022)
- [Impossibility Results in AI: A Survey](https://mlsanity.com/paper/2109.00484/) (2021)
- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Explicit Explore, Exploit, or Escape (E4): near-optimal safety-constrained reinforcement learning in polynomial time](https://mlsanity.com/paper/2111.07395/) (2021)
- [Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impact](https://mlsanity.com/paper/2507.00951/) (2025)
- [Should attention be all we need? The epistemic and ethical implications of unification in machine learning](https://mlsanity.com/paper/2205.08377/) (2022)
- [Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment](https://mlsanity.com/paper/2406.11039/) (2024)
- [Counterfactual Planning in AGI Systems](https://mlsanity.com/paper/2102.00834/) (2021)
- [Why do Experts Disagree on Existential Risk and P(doom)? A Survey of AI Experts](https://mlsanity.com/paper/2502.14870/) (2025)
- [AI Risk Skepticism](https://mlsanity.com/paper/2105.02704/) (2021)
- [Technology and Consciousness](https://mlsanity.com/paper/2209.03956/) (2022)
- [The Promise and Peril of Artificial Intelligence -- Violet Teaming Offers a Balanced Path Forward](https://mlsanity.com/paper/2308.14253/) (2023)
- [The Elephant in the Room -- Why AI Safety Demands Diverse Teams](https://mlsanity.com/paper/2407.10254/) (2024)
- [Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization](https://mlsanity.com/paper/2503.07660/) (2025)
- [Winning at All Cost: A Small Environment for Eliciting Specification Gaming Behaviors in Large Language Models](https://mlsanity.com/paper/2505.07846/) (2025)
- [Exploring the Constraints on Artificial General Intelligence: A Game-Theoretic No-Go Theorem](https://mlsanity.com/paper/2209.12346/) (2022)
- [Towards Integrated Alignment](https://mlsanity.com/paper/2508.06592/) (2025)
- [Individual utilities of life satisfaction reveal inequality aversion unrelated to political alignment](https://mlsanity.com/paper/2509.07793/) (2025)
- [Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirements](https://mlsanity.com/paper/2605.12963/) (2026)

## Abstract (arXiv)

The development of Artificial General Intelligence (AGI) promises to be a major event. Along with its many potential benefits, it also raises serious safety concerns (Bostrom, 2014). The intention of this paper is to provide an easily accessible and up-to-date collection of references for the emerging field of AGI safety. A significant number of safety problems for AGI have been identified. We list these, and survey recent research on solving them. We also cover works on how best to think of AGI from the limited knowledge we have today, predictions for when AGI will first be created, and what will happen after its creation. Finally, we review the current public policy on AGI.