# An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1806.03820 · HTML: https://mlsanity.com/paper/1806.03820/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 11 Jun 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 5 of 5)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Pragmatic-Pedagogic Value Alignment](https://mlsanity.com/paper/1707.06354/) (2017)

## Built on by (most cited first, 15 of 15)

- [A Survey of Ad Hoc Teamwork Research](https://mlsanity.com/paper/2202.10450/) (2022)
- [AssistanceZero: Scalably Solving Assistance Games](https://mlsanity.com/paper/2504.07091/) (2025)
- [A Review on Machine Theory of Mind](https://mlsanity.com/paper/2303.11594/) (2023)
- [Non-Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1911.04220/) (2019)
- [Active teacher selection for reward learning](https://mlsanity.com/paper/2310.15288/) (2023)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy](https://mlsanity.com/paper/2510.26752/) (2025)
- [Solving Hierarchical Information-Sharing Dec-POMDPs: An Extensive-Form Game Approach](https://mlsanity.com/paper/2402.02954/) (2024)
- [Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory](https://mlsanity.com/paper/2405.19024/) (2024)
- [Reward Shaping for Human Learning via Inverse Reinforcement Learning](https://mlsanity.com/paper/2002.10904/) (2020)
- [Balancing Performance and Human Autonomy with Implicit Guidance Agent](https://mlsanity.com/paper/2109.00414/) (2021)
- [Online Learning of Human Constraints from Feedback in Shared Autonomy](https://mlsanity.com/paper/2403.02974/) (2024)
- [Supervised Reward Inference](https://mlsanity.com/paper/2502.18447/) (2025)
- [Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2602.08835/) (2026)
- [Provably Optimal Learning Algorithms for Assistance Games](https://mlsanity.com/paper/2607.08012/) (2026)

## Abstract (arXiv)

Our goal is for AI systems to correctly identify and act according to their human user's objectives. Cooperative Inverse Reinforcement Learning (CIRL) formalizes this value alignment problem as a two-player game between a human and robot, in which only the human knows the parameters of the reward function: the robot needs to learn them as the interaction unfolds. Previous work showed that CIRL can be solved as a POMDP, but with an action space size exponential in the size of the reward parameter space. In this work, we exploit a specific property of CIRL---the human is a full information agent---to derive an optimality-preserving modification to the standard Bellman update; this reduces the complexity of the problem by an exponential factor and allows us to relax CIRL's assumption of human rationality. We apply this update to a variety of POMDP solvers and find that it enables us to scale CIRL to non-trivial problems, with larger reward parameter spaces, and larger action spaces for both robot and human. In solutions to these larger problems, the human exhibits pedagogic (teaching) behavior, while the robot interprets it as such and attains higher value for the human.