# AI Safety: A Climb To Armageddon?

arXiv: https://arxiv.org/abs/2405.19832 · HTML: https://mlsanity.com/paper/2405.19832/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 6 of 6)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation](https://mlsanity.com/paper/1802.07228/) (2018)
- [Guidelines for Artificial Intelligence Containment](https://mlsanity.com/paper/1707.08476/) (2017)

## Built on by (most cited first, 2 of 2)

- [AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?](https://mlsanity.com/paper/2510.11235/) (2025)
- [Do LLMs have core beliefs?](https://mlsanity.com/paper/2605.03255/) (2026)

## Abstract (arXiv)

This paper presents an argument that certain AI safety measures, rather than mitigating existential risk, may instead exacerbate it. Under certain key assumptions - the inevitability of AI failure, the expected correlation between an AI system's power at the point of failure and the severity of the resulting harm, and the tendency of safety measures to enable AI systems to become more powerful before failing - safety efforts have negative expected utility. The paper examines three response strategies: Optimism, Mitigation, and Holism. Each faces challenges stemming from intrinsic features of the AI safety landscape that we term Bottlenecking, the Perfection Barrier, and Equilibrium Fluctuation. The surprising robustness of the argument forces a re-examination of core assumptions around AI safety and points to several avenues for further research.