# The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes

arXiv: https://arxiv.org/abs/2603.20994 · HTML: https://mlsanity.com/paper/2603.20994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning](https://mlsanity.com/paper/2412.13631/) (2024)
- [Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates](https://mlsanity.com/paper/2506.22276/) (2025)
- [Rebellion and Disobedience as Useful Tools in Human-Robot Interaction Research -- The Handheld Robotics Case](https://mlsanity.com/paper/2205.03968/) (2022)
- [Shared Control with Black Box Agents using Oracle Queries](https://mlsanity.com/paper/2410.19612/) (2024)

## Abstract (arXiv)

In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as ``safety traps,'' where the system indefinitely avoids harm but fails to achieve the human's goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents.