# Should Robots be Obedient?

arXiv: https://arxiv.org/abs/1705.09990 · HTML: https://mlsanity.com/paper/1705.09990/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2017.

## Built on by (most cited first, 20 of 20)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [GR-3 Technical Report](https://mlsanity.com/paper/2507.15493/) (2025)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Why human-AI relationships need socioaffective alignment](https://mlsanity.com/paper/2502.02528/) (2025)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [Hard Choices in Artificial Intelligence](https://mlsanity.com/paper/2106.11022/) (2021)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [LLM Theory of Mind and Alignment: Opportunities and Risks](https://mlsanity.com/paper/2405.08154/) (2024)
- [Inferring the Goals of Communicating Agents from Actions and Instructions](https://mlsanity.com/paper/2306.16207/) (2023)
- [AI Risk Management Should Incorporate Both Safety and Security](https://mlsanity.com/paper/2405.19524/) (2024)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [A Complete Criterion for Value of Information in Soluble Influence Diagrams](https://mlsanity.com/paper/2202.11629/) (2022)
- [Auditing Robot Learning for Safety and Compliance during Deployment](https://mlsanity.com/paper/2110.05702/) (2021)
- [Off-Switching Not Guaranteed](https://mlsanity.com/paper/2502.08864/) (2025)
- [Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch](https://mlsanity.com/paper/2404.08791/) (2024)
- [Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?](https://mlsanity.com/paper/2505.13257/) (2025)
- [Password-Activated Shutdown Protocols for Misaligned Frontier Agents](https://mlsanity.com/paper/2512.03089/) (2025)
- [Shutdown Safety Valves for Advanced AI](https://mlsanity.com/paper/2603.07315/) (2026)
- [Towards Responsibly Non-Compliant Machines](https://mlsanity.com/paper/2606.12147/) (2026)
- [Corrigible Assistance in One Round: Pragmatic-Pedagogic Best Response](https://mlsanity.com/paper/2607.27508/) (2026)

## Abstract (arXiv)

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.