mlsanityData to 7 Oct 2026

Learning to Influence Human Behavior with Offline Reinforcement Learning

arXiv 2303.02265 ↗First posted 3 Mar 2023cs.AI cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

When interacting with people, AI agents do not just influence the state of the world -- they also influence the actions people take in response to the agent, and even their underlying intentions and strategies. Accounting for and leveraging this influence has mostly been studied in settings where it is sufficient to assume that human behavior is near-optimal: competitive games, or general-sum settings like autonomous driving alongside human drivers. Instead, we focus on influence in settings where there is a need to capture human suboptimality. For instance, imagine a collaborative task in which, due either to cognitive biases or lack of information, people do not perform very well -- how could an agent influence them towards more optimal behavior? Assuming near-optimal human behavior will not work here, and so the agent needs to learn from real human data. But experimenting online with humans is potentially unsafe, and creating a high-fidelity simulator of the environment is often impractical. Hence, we focus on learning from an offline dataset of human-human interactions. Our observation is that offline reinforcement learning (RL) can learn to effectively influence suboptimal humans by extending and combining elements of observed human-human behavior. We demonstrate that offline RL can solve two challenges with effective influence. First, we show that by learning from a dataset of suboptimal human-human interaction on a variety of tasks -- none of which contains examples of successful influence -- an agent can learn influence strategies to steer humans towards better performance even on new tasks. Second, we show that by also modeling and conditioning on human behavior, offline RL can learn to affect not just the human's actions but also their underlying strategy, and adapt to changes in their strategy.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 36 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

26 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models22 Apr

Other references · 25

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2016Deep Variational Information Bottleneckcited by 915
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  9. 2017Population Based Training of Neural Networkscited by 351
  10. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  11. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  12. 2017Learning with Opponent-Learning Awarenesscited by 218
13 more
  1. 2021Reinforcement learning based recommender systems: A surveycited by 187
  2. 2021Collaborating with Humans without Human Datacited by 139
  3. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  4. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
  5. 2020Learning Latent Representations to Influence Multi-Agent Interactioncited by 92
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  7. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  8. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  9. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
  10. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  11. 2021Influencing Towards Stable Multi-Agent Interactionscited by 24
  12. 2022Influencing Long-Term Behavior in Multiagent Reinforcement Learningcited by 18
  13. 2019Learning from My Partner's Actions: Roles in Decentralized Robot Teamscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  2. 2026CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning
  2. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information
  3. 2021Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations
  4. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2023Learning Complementary Policies for Human-AI Teams
  7. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  8. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback