# Learning to Cooperate with Humans using Generative Agents

arXiv: https://arxiv.org/abs/2411.13934 · HTML: https://mlsanity.com/paper/2411.13934/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Deep reinforcement learning methods / [Multi-agent reinforcement learning](https://mlsanity.com/deeprl/multi-agent-reinforcement-learning/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [RL for Latent MDPs: Regret Guarantees and a Lower Bound](https://mlsanity.com/paper/2102.04939/) (2021)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [Agent Modelling under Partial Observability for Deep Reinforcement Learning](https://mlsanity.com/paper/2006.09447/) (2020)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1912.02288/) (2019)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [Theory of Minds: Understanding Behavior in Groups Through Inverse Planning](https://mlsanity.com/paper/1901.06085/) (2019)
- [Cooperative Open-ended Learning Framework for Zero-shot Coordination](https://mlsanity.com/paper/2302.04831/) (2023)
- [PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2301.06387/) (2023)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [Diverse Conventions for Human-AI Collaboration](https://mlsanity.com/paper/2310.15414/) (2023)
- [Coordination with Humans via Strategy Matching](https://mlsanity.com/paper/2210.15099/) (2022)

## Built on by (most cited first, 24 of 24)

- [Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination](https://mlsanity.com/paper/2504.12714/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)](https://mlsanity.com/paper/2505.17323/) (2025)
- [Adaptively Coordinating with Novel Partners via Learned Latent Strategies](https://mlsanity.com/paper/2511.12754/) (2025)
- [Improving Human-AI Coordination through Online Adversarial Training and Generative Models](https://mlsanity.com/paper/2504.15457/) (2025)
- [Unsupervised Partner Design Enables Robust Ad-hoc Teamwork](https://mlsanity.com/paper/2508.06336/) (2025)
- [Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractions](https://mlsanity.com/paper/2505.04579/) (2025)
- [Behavioral Generative Agents for Energy Operations](https://mlsanity.com/paper/2506.12664/) (2025)
- [Robust and Diverse Multi-Agent Learning via Rational Policy Gradient](https://mlsanity.com/paper/2511.09535/) (2025)
- [Training Generalizable Collaborative Agents via Strategic Risk Aversion](https://mlsanity.com/paper/2602.21515/) (2026)
- [Shaping Zero-Shot Coordination via State Blocking](https://mlsanity.com/paper/2605.11688/) (2026)
- [Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork](https://mlsanity.com/paper/2605.24423/) (2026)
- [Algorithmic Prompt Generation for Diverse Human-like Teaming and Communication with Large Language Models](https://mlsanity.com/paper/2504.03991/) (2025)
- [Modeling Latent Partner Strategies for Adaptive Zero-Shot Human-Agent Collaboration](https://mlsanity.com/paper/2507.05244/) (2025)
- [Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2609.14896/) (2026)
- [Towards Zero-Shot Coordination between Teams of Agents: The N-XPlay Framework](https://mlsanity.com/paper/2506.17560/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Moving Out: Physically-grounded Human-AI Collaboration](https://mlsanity.com/paper/2507.18623/) (2025)
- [Generative Modeling for Robust Deep Reinforcement Learning on the Traveling Salesman Problem](https://mlsanity.com/paper/2508.08718/) (2025)
- [Beyond Partner Diversity: An Influence-Based Team Steering Framework for Zero-Shot Human-Machine Teaming](https://mlsanity.com/paper/2605.15400/) (2026)
- [Dreaming Of Others: Latent Teammate Modeling In World Models For Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.31361/) (2026)
- [MindZero: Learning Online Mental Reasoning With Zero Annotations](https://mlsanity.com/paper/2606.00240/) (2026)
- [Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization](https://mlsanity.com/paper/2606.05602/) (2026)
- [PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination](https://mlsanity.com/paper/2610.15969/) (2026)

## Abstract (arXiv)

Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show learning a generative model of human partners can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method -- Generative Agent Modeling for Multi-agent Adaptation (GAMMA) -- on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.