# The Assistive Multi-Armed Bandit

arXiv: https://arxiv.org/abs/1901.08654 · HTML: https://mlsanity.com/paper/1901.08654/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2019.

## Built on by (most cited first, 19 of 19)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [Towards Modeling and Influencing the Dynamics of Human Learning](https://mlsanity.com/paper/2301.00901/) (2023)
- [A Bayesian Approach to Robust Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.08571/) (2023)
- [Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits](https://mlsanity.com/paper/2106.14866/) (2021)
- [Active Inference for Autonomous Decision-Making with Contextual Multi-Armed Bandits](https://mlsanity.com/paper/2209.09185/) (2022)
- [AI-Assisted Decision Making with Human Learning](https://mlsanity.com/paper/2502.13062/) (2025)
- [When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making](https://mlsanity.com/paper/2308.11721/) (2023)
- [Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2410.13957/) (2024)
- [Collaborative Intelligence in Sequential Experiments: A Human-in-the-Loop Framework for Drug Discovery](https://mlsanity.com/paper/2405.03942/) (2024)
- [Multi-Round Human-AI Collaboration with User-Specified Requirements](https://mlsanity.com/paper/2602.17646/) (2026)
- [Necessary and Sufficient Conditions for Inverse Reinforcement Learning of Bayesian Stopping Time Problems](https://mlsanity.com/paper/2007.03481/) (2020)
- [Effective Dimension in Bandit Problems under Censorship](https://mlsanity.com/paper/2302.06916/) (2023)
- [Strategic Decision Support for AI Agents](https://mlsanity.com/paper/2606.12587/) (2026)
- [Multi-armed Bandit Algorithms on System-on-Chip: Go Frequentist or Bayesian?](https://mlsanity.com/paper/2106.02855/) (2021)
- [Learning the Preferences of a Learning Agent](https://mlsanity.com/paper/2605.09217/) (2026)

## Abstract (arXiv)

Learning preferences implicit in the choices humans make is a well studied problem in both economics and computer science. However, most work makes the assumption that humans are acting (noisily) optimally with respect to their preferences. Such approaches can fail when people are themselves learning about what they want. In this work, we introduce the assistive multi-armed bandit, where a robot assists a human playing a bandit task to maximize cumulative reward. In this problem, the human does not know the reward function but can learn it through the rewards received from arm pulls; the robot only observes which arms the human pulls but not the reward associated with each pull. We offer sufficient and necessary conditions for successfully assisting the human in this framework. Surprisingly, better human performance in isolation does not necessarily lead to better performance when assisted by the robot: a human policy can do better by effectively communicating its observed rewards to the robot. We conduct proof-of-concept experiments that support these results. We see this work as contributing towards a theory behind algorithms for human-robot interaction.