# Multi-Principal Assistance Games

arXiv: https://arxiv.org/abs/2007.09540 · HTML: https://mlsanity.com/paper/2007.09540/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 19 Jul 2020.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Built on by (most cited first, 10 of 10)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [Designing Fiduciary Artificial Intelligence](https://mlsanity.com/paper/2308.02435/) (2023)
- [Multi-User Large Language Model Agents](https://mlsanity.com/paper/2604.08567/) (2026)
- [Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction](https://mlsanity.com/paper/2607.00001/) (2026)
- [Computability, Complexity, Consistency and Controllability: A Four C's Framework for cross-disciplinary Ethical Algorithm Research](https://mlsanity.com/paper/2102.04234/) (2021)
- [Dynamic value alignment through preference aggregation of multiple objectives](https://mlsanity.com/paper/2310.05871/) (2023)
- [LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback](https://mlsanity.com/paper/2607.29559/) (2026)

## Abstract (arXiv)

Assistance games (also known as cooperative inverse reinforcement learning games) have been proposed as a model for beneficial AI, wherein a robotic agent must act on behalf of a human principal but is initially uncertain about the humans payoff function. This paper studies multi-principal assistance games, which cover the more general case in which the robot acts on behalf of N humans who may have widely differing payoffs. Impossibility theorems in social choice theory and voting theory can be applied to such games, suggesting that strategic behavior by the human principals may complicate the robots task in learning their payoffs. We analyze in particular a bandit apprentice game in which the humans act first to demonstrate their individual preferences for the arms and then the robot acts to maximize the sum of human payoffs. We explore the extent to which the cost of choosing suboptimal arms reduces the incentive to mislead, a form of natural mechanism design. In this context we propose a social choice method that uses shared control of a system to combine preference inference with social welfare optimization.