# Incorrigibility in the CIRL Framework

arXiv: https://arxiv.org/abs/1709.06275 · HTML: https://mlsanity.com/paper/1709.06275/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2017.

## Built on by (most cited first, 9 of 9)

- [TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI](https://mlsanity.com/paper/2306.06924/) (2023)
- [The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists](https://mlsanity.com/paper/2403.04471/) (2024)
- [Impossibility Results in AI: A Survey](https://mlsanity.com/paper/2109.00484/) (2021)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Axes for Sociotechnical Inquiry in AI Research](https://mlsanity.com/paper/2105.06551/) (2021)
- [The Partially Observable Off-Switch Game](https://mlsanity.com/paper/2411.17749/) (2024)
- [Off-Switching Not Guaranteed](https://mlsanity.com/paper/2502.08864/) (2025)
- [Password-Activated Shutdown Protocols for Misaligned Frontier Agents](https://mlsanity.com/paper/2512.03089/) (2025)
- [Shutdown Safety Valves for Advanced AI](https://mlsanity.com/paper/2603.07315/) (2026)

## Abstract (arXiv)

A value learning system has incentives to follow shutdown instructions, assuming the shutdown instruction provides information (in the technical sense) about which actions lead to valuable outcomes. However, this assumption is not robust to model mis-specification (e.g., in the case of programmer errors). We demonstrate this by presenting some Supervised POMDP scenarios in which errors in the parameterized reward function remove the incentive to follow shutdown commands. These difficulties parallel those discussed by Soares et al. (2015) in their paper on corrigibility. We argue that it is important to consider systems that follow shutdown commands under some weaker set of assumptions (e.g., that one small verified module is correctly implemented; as opposed to an entire prior probability distribution and/or parameterized reward function). We discuss some difficulties with simple ways to attempt to attain these sorts of guarantees in a value learning framework.