# Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurs

arXiv: https://arxiv.org/abs/2204.01160 · HTML: https://mlsanity.com/paper/2204.01160/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 44 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 7 of 7)

- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Cognitive science as a source of forward and inverse models of human decisions for robotics and control](https://mlsanity.com/paper/2109.00127/) (2021)
- [Tractable Inference for Complex Stochastic Processes](https://mlsanity.com/paper/1301.7362/) (2013)
- [Learning in POMDPs with Monte Carlo Tree Search](https://mlsanity.com/paper/1806.05631/) (2018)
- [Time-Inconsistent Planning: A Computational Problem in Behavioral Economics](https://mlsanity.com/paper/1405.1254/) (2014)
- [Planning Problems for Sophisticated Agents with Present Bias](https://mlsanity.com/paper/1603.08177/) (2016)

## Built on by (most cited first, 5 of 5)

- [AI for Science: An Emerging Agenda](https://mlsanity.com/paper/2303.04217/) (2023)
- [Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory](https://mlsanity.com/paper/2405.19024/) (2024)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Uncoupled Learning of Differential Stackelberg Equilibria with Commitments](https://mlsanity.com/paper/2302.03438/) (2023)
- [Propose to Learn, Learn to Propose: Evaluability-Aware Assistance under Bounded Rationality](https://mlsanity.com/paper/2609.02242/) (2026)

## Abstract (arXiv)

Centaurs are half-human, half-AI decision-makers where the AI's goal is to complement the human. To do so, the AI must be able to recognize the goals and constraints of the human and have the means to help them. We present a novel formulation of the interaction between the human and the AI as a sequential game where the agents are modelled using Bayesian best-response models. We show that in this case the AI's problem of helping bounded-rational humans make better decisions reduces to a Bayes-adaptive POMDP. In our simulated experiments, we consider an instantiation of our framework for humans who are subjectively optimistic about the AI's future behaviour. Our results show that when equipped with a model of the human, the AI can infer the human's bounds and nudge them towards better decisions. We discuss ways in which the machine can learn to improve upon its own limitations as well with the help of the human. We identify a novel trade-off for centaurs in partially observable tasks: for the AI's actions to be acceptable to the human, the machine must make sure their beliefs are sufficiently aligned, but aligning beliefs might be costly. We present a preliminary theoretical analysis of this trade-off and its dependence on task structure.