# Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Tasks

arXiv: https://arxiv.org/abs/2401.14923 · HTML: https://mlsanity.com/paper/2401.14923/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 26 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 13 of 13)

- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Plannable Approximations to MDP Homomorphisms: Equivariance under Actions](https://mlsanity.com/paper/2002.11963/) (2020)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference](https://mlsanity.com/paper/1906.09624/) (2019)
- [Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines](https://mlsanity.com/paper/2206.03944/) (2022)
- [Sample-efficient Reinforcement Learning in Robotic Table Tennis](https://mlsanity.com/paper/2011.03275/) (2020)
- [Assisted Perception: Optimizing Observations to Communicate State](https://mlsanity.com/paper/2008.02840/) (2020)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Sample-efficient Deep Reinforcement Learning with Imaginary Rollouts for Human-Robot Interaction](https://mlsanity.com/paper/1908.05546/) (2019)
- [Modeling and Interpreting Real-world Human Risk Decision Making with Inverse Reinforcement Learning](https://mlsanity.com/paper/1906.05803/) (2019)
- [Using Cognitive Models to Train Warm Start Reinforcement Learning Agents for Human-Computer Interactions](https://mlsanity.com/paper/2103.06160/) (2021)

## Built on by (most cited first, 2 of 2)

- [Optimizing Delegation in Collaborative Human-AI Hybrid Teams](https://mlsanity.com/paper/2402.05605/) (2024)
- [Improving Human Performance with Value-Aware Interventions: A Case Study in Chess](https://mlsanity.com/paper/2604.14465/) (2026)

## Abstract (arXiv)

Many important behavior changes are frictionful; they require individuals to expend effort over a long period with little immediate gratification. Here, an artificial intelligence (AI) agent can provide personalized interventions to help individuals stick to their goals. In these settings, the AI agent must personalize rapidly (before the individual disengages) and interpretably, to help us understand the behavioral interventions. In this paper, we introduce Behavior Model Reinforcement Learning (BMRL), a framework in which an AI agent intervenes on the parameters of a Markov Decision Process (MDP) belonging to a boundedly rational human agent. Our formulation of the human decision-maker as a planning agent allows us to attribute undesirable human policies (ones that do not lead to the goal) to their maladapted MDP parameters, such as an extremely low discount factor. Furthermore, we propose a class of tractable human models that captures fundamental behaviors in frictionful tasks. Introducing a notion of MDP equivalence specific to BMRL, we theoretically and empirically show that AI planning with our human models can lead to helpful policies on a wide range of more complex, ground-truth humans.