mlsanityData to 7 Oct 2026

The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists

arXiv 2403.04471 ↗First posted 7 Mar 2024cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

I explain the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don't try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems show that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it's costly to do so. And patience trades off against shutdownability: the more patient an agent, the greater the costs that agent is willing to incur to manipulate the shutdown button. I end by noting that these theorems can guide our search for solutions.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 20 of its 49 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

20 of 49 references.

References · 20

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  7. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  8. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  9. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  10. 2019Optimal Policies Tend to Seek Powercited by 115
  11. 2017AI Safety Gridworldscited by 108
  12. 2016The Off-Switch Gamecited by 97
8 more
  1. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  2. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  3. 2023Eight Things to Know about Large Language Modelscited by 73
  4. 2019Conservative Agency via Attainable Utility Preservationcited by 35
  5. 2023Human Control: Definitions and Algorithmscited by 10
  6. 2017Incorrigibility in the CIRL Frameworkcited by 9
  7. 2017A Game-Theoretic Analysis of the Off-Switch Gamecited by 8
  8. 2023Evaluating Shutdown Avoidance of Language Models in Textual Scenarioscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  2. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  3. 2024Beyond Preferences in AI Alignmentcited by 46
  4. 2024Frontier AI systems have surpassed the self-replicating red linecited by 16
  5. 2025AI Testing Should Account for Sophisticated Strategic Behaviourcited by 6
  6. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  7. 2026Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systemscited by 4
  8. 2025Large language model-powered AI systems achieve self-replication with no human interventioncited by 3
  9. 2024Towards Shutdownable Agents via Stochastic Choicecited by 2
  10. 2025What Is AI Safety? What Do We Want It to Be?cited by 2
  11. 2025Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscapecited by 2
  12. 2026Quantifying Self-Preservation Bias in Large Language Modelscited by 2
10 more
  1. 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviorscited by 2
  2. 2026Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMscited by 1
  3. 2026Eigenism: Ethics for a Human-AI Futurecited by 1
  4. 2024Evaluating Stability of Unreflective Alignmentcited by 0
  5. 2025Comparative Analysis of Large Language Models for the Machine-Assisted Resolution of User Intentionscited by 0
  6. 2025Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibilitycited by 0
  7. 2026AI Survival Stories: a Taxonomic Analysis of AI Existential Riskcited by 0
  8. 2026Questionnaire Responses Do not Capture the Safety of AI Agentscited by 0
  9. 2026When AI Says It Feelscited by 0
  10. 2026Revisiting the shutdown problemcited by 0