# The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists

arXiv: https://arxiv.org/abs/2403.04471 · HTML: https://mlsanity.com/paper/2403.04471/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents](https://mlsanity.com/paper/2401.12963/) (2024)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Conservative Agency via Attainable Utility Preservation](https://mlsanity.com/paper/1902.09725/) (2019)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Incorrigibility in the CIRL Framework](https://mlsanity.com/paper/1709.06275/) (2017)
- [A Game-Theoretic Analysis of the Off-Switch Game](https://mlsanity.com/paper/1708.03871/) (2017)
- [Evaluating Shutdown Avoidance of Language Models in Textual Scenarios](https://mlsanity.com/paper/2307.00787/) (2023)

## Built on by (most cited first, 22 of 22)

- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Frontier AI systems have surpassed the self-replicating red line](https://mlsanity.com/paper/2412.12140/) (2024)
- [AI Testing Should Account for Sophisticated Strategic Behaviour](https://mlsanity.com/paper/2508.14927/) (2025)
- [Large language model-powered AI systems achieve self-replication with no human intervention](https://mlsanity.com/paper/2503.17378/) (2025)
- [Misalignment or misuse? The AGI alignment tradeoff](https://mlsanity.com/paper/2506.03755/) (2025)
- [Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systems](https://mlsanity.com/paper/2601.00360/) (2026)
- [Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape](https://mlsanity.com/paper/2507.07765/) (2025)
- [Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors](https://mlsanity.com/paper/2605.06490/) (2026)
- [Towards Shutdownable Agents via Stochastic Choice](https://mlsanity.com/paper/2407.00805/) (2024)
- [What Is AI Safety? What Do We Want It to Be?](https://mlsanity.com/paper/2505.02313/) (2025)
- [Quantifying Self-Preservation Bias in Large Language Models](https://mlsanity.com/paper/2604.02174/) (2026)
- [Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs](https://mlsanity.com/paper/2604.17502/) (2026)
- [Eigenism: Ethics for a Human-AI Future](https://mlsanity.com/paper/2606.12420/) (2026)
- [Evaluating Stability of Unreflective Alignment](https://mlsanity.com/paper/2408.15116/) (2024)
- [Comparative Analysis of Large Language Models for the Machine-Assisted Resolution of User Intentions](https://mlsanity.com/paper/2510.08576/) (2025)
- [Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibility](https://mlsanity.com/paper/2511.13725/) (2025)
- [AI Survival Stories: a Taxonomic Analysis of AI Existential Risk](https://mlsanity.com/paper/2601.09765/) (2026)
- [Questionnaire Responses Do not Capture the Safety of AI Agents](https://mlsanity.com/paper/2603.14417/) (2026)
- [When AI Says It Feels](https://mlsanity.com/paper/2606.05734/) (2026)
- [Revisiting the shutdown problem](https://mlsanity.com/paper/2606.08296/) (2026)

## Abstract (arXiv)

I explain the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don't try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems show that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it's costly to do so. And patience trades off against shutdownability: the more patient an agent, the greater the costs that agent is willing to incur to manipulate the shutdown button. I end by noting that these theorems can guide our search for solutions.