# The AI off-switch problem as a signalling game: bounded rationality and incomparability

arXiv: https://arxiv.org/abs/2502.06403 · HTML: https://mlsanity.com/paper/2502.06403/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 48 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 8 of 8)

- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Preferential Bayesian optimisation with Skew Gaussian Processes](https://mlsanity.com/paper/2008.06677/) (2020)
- [A Game-Theoretic Analysis of the Off-Switch Game](https://mlsanity.com/paper/1708.03871/) (2017)
- [Skew Gaussian Processes for Classification](https://mlsanity.com/paper/2005.12987/) (2020)
- [Archimedean Choice Functions: an Axiomatic Foundation for Imprecise Decision Making](https://mlsanity.com/paper/2002.05196/) (2020)
- [A unified framework for closed-form nonparametric regression, classification, preference and mixed problems with Skew Gaussian Processes](https://mlsanity.com/paper/2012.06846/) (2020)
- [Learning Choice Functions with Gaussian Processes](https://mlsanity.com/paper/2302.00406/) (2023)
- [A tutorial on learning from preferences and choices with Gaussian Processes](https://mlsanity.com/paper/2403.11782/) (2024)

## Abstract (arXiv)

The off-switch problem is a critical challenge in AI control: if an AI system resists being switched off, it poses a significant risk. In this paper, we model the off-switch problem as a signalling game, where a human decision-maker communicates its preferences about some underlying decision problem to an AI agent, which then selects actions to maximise the human's utility. We assume that the human is a bounded rational agent and explore various bounded rationality mechanisms. Using real machine learning models, we reprove prior results and demonstrate that a necessary condition for an AI system to refrain from disabling its off-switch is its uncertainty about the human's utility. We also analyse how message costs influence optimal strategies and extend the analysis to scenarios involving incomparability.