mlsanityData to 7 Oct 2026

The Off-Switch Game

arXiv 1611.08219 ↗First posted 24 Nov 2016cs.AIcited by 97

Not on the map. Why?

Abstract (arXiv)

It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off. We analyze a simple game between a human H and a robot R, where H can press R's off switch but R can disable the off switch. A traditional agent takes its reward function for granted: we show that such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility. (R also has no incentive to switch itself off in this setting.) We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 0; 2021: 12; 2022: 6; 2023: 9; 2024: 8; 2025: 28; 2026: 32.

97 papers.

Sort:

By area:

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  4. 2023An Overview of Catastrophic AI Riskscited by 214
  5. 2022Defining and Characterizing Reward Hackingcited by 184
  6. 2021Alignment of Language Agentscited by 167
  7. 2023Model evaluation for extreme riskscited by 167
  8. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  9. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  10. 2021Consequences of Misaligned AIcited by 79
  11. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  12. 2021On the Expressivity of Markov Rewardcited by 71
38 more
  1. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  2. 2022X-Risk Analysis for AI Researchcited by 60
  3. 2021Agent Incentives: A Causal Perspectivecited by 51
  4. 2023Managing extreme AI risks amid rapid progresscited by 48
  5. 2024Beyond Preferences in AI Alignmentcited by 46
  6. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  7. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  8. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  9. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  10. 2023The Machine Psychology of Cooperation: Can GPT models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games?cited by 22
  11. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  12. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  13. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  14. 2021Impossibility Results in AI: A Surveycited by 13
  15. 2024Contestable AI needs Computational Argumentationcited by 12
  16. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  17. 2023Human Control: Definitions and Algorithmscited by 10
  18. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 7
  19. 2026Right to History: A Sovereignty Kernel for Verifiable AI Agent Executioncited by 7
  20. 2025Agentic Services Computingcited by 6
  21. 2025Harmful Traits of AI Companionscited by 6
  22. 2021Axes for Sociotechnical Inquiry in AI Researchcited by 5
  23. 2025Emergent Risk Awareness in Rational Agents under Resource Constraintscited by 5
  24. 2025Enhancing LLM Agent Safety via Causal Influence Promptingcited by 5
  25. 2024Observation Interference in Partially Observable Assistance Gamescited by 4
  26. 2025Off-Switching Not Guaranteedcited by 4
  27. 2025Core Safety Values for Provably Corrigible Agentscited by 4
  28. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 4
  29. 2021Trust-Aware Planning: Modeling Trust Evolution in Iterated Human-Robot Interactioncited by 3
  30. 2023Machine Lovecited by 3
  31. 2025Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Makingcited by 3
  32. 2025VerificAgent: Domain-Specific Memory Verification for Scalable Oversight of Aligned Computer-Use Agentscited by 3
  33. 2025The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomycited by 3
  34. 2024Towards Shutdownable Agents via Stochastic Choicecited by 2
  35. 2025Why do Experts Disagree on Existential Risk and P(doom)? A Survey of AI Expertscited by 2
  36. 2025AI Agents Should be Regulated Based on the Extent of Their Autonomous Operationscited by 2
  37. 2025Misalignment from Treating Means as Endscited by 2
  38. 2025Learning "Partner-Aware" Collaborators in Multi-Party Collaborationcited by 2

50 most cited of 97 All 97 (JSON)