# Human Control: Definitions and Algorithms

arXiv: https://arxiv.org/abs/2305.19861 · HTML: https://mlsanity.com/paper/2305.19861/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 20 of 20)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Conservative Agency via Attainable Utility Preservation](https://mlsanity.com/paper/1902.09725/) (2019)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Avoiding Side Effects By Considering Future Tasks](https://mlsanity.com/paper/2010.07877/) (2020)
- [Counterfactual harm](https://mlsanity.com/paper/2204.12993/) (2022)
- [Should Robots be Obedient?](https://mlsanity.com/paper/1705.09990/) (2017)
- [Path-Specific Objectives for Safer Agent Incentives](https://mlsanity.com/paper/2204.10018/) (2022)
- [Death and Suicide in Universal Artificial Intelligence](https://mlsanity.com/paper/1606.00652/) (2016)
- [Incorrigibility in the CIRL Framework](https://mlsanity.com/paper/1709.06275/) (2017)
- [Corrigibility with Utility Preservation](https://mlsanity.com/paper/1908.01695/) (2019)
- [Definitions of intent suitable for algorithms](https://mlsanity.com/paper/2106.04235/) (2021)
- ['Indifference' methods for managing agent rewards](https://mlsanity.com/paper/1712.06365/) (2017)

## Built on by (most cited first, 10 of 10)

- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation](https://mlsanity.com/paper/2406.11852/) (2024)
- [The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists](https://mlsanity.com/paper/2403.04471/) (2024)
- [Emergent Risk Awareness in Rational Agents under Resource Constraints](https://mlsanity.com/paper/2505.23436/) (2025)
- [Quantifying stability of non-power-seeking in artificial agents](https://mlsanity.com/paper/2401.03529/) (2024)
- [Towards Shutdownable Agents via Stochastic Choice](https://mlsanity.com/paper/2407.00805/) (2024)
- [Golden Handcuffs make safer AI agents](https://mlsanity.com/paper/2604.13609/) (2026)
- [Password-Activated Shutdown Protocols for Misaligned Frontier Agents](https://mlsanity.com/paper/2512.03089/) (2025)
- [Shutdown Safety Valves for Advanced AI](https://mlsanity.com/paper/2603.07315/) (2026)

## Abstract (arXiv)

How can humans stay in control of advanced artificial intelligence systems? One proposal is corrigibility, which requires the agent to follow the instructions of a human overseer, without inappropriately influencing them. In this paper, we formally define a variant of corrigibility called shutdown instructability, and show that it implies appropriate shutdown behavior, retention of human autonomy, and avoidance of user harm. We also analyse the related concepts of non-obstruction and shutdown alignment, three previously proposed algorithms for human control, and one new algorithm.