mlsanityData to 14 Oct 2026

On the Utility of Learning about Humans for Human-AI Coordination

arXiv 1910.05789 ↗First posted 13 Oct 2019cs.LG cs.AI cs.HCcited by 349

Not on the map. Why?

Cited by 63 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https://github.com/HumanCompatibleAI/overcooked_ai.

Key facts. Cited by 349 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 39; 2023: 61; 2024: 65; 2025: 98; 2026: 60.

349 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  3. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  4. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  5. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  6. 2021Collaborating with Humans without Human Datacited by 142
  7. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  8. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  9. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  10. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  11. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 98
  12. 2023MindAgent: Emergent Gaming Interactioncited by 88
38 more
  1. 2024Simplifying Deep Temporal Difference Learningcited by 78
  2. 2021Off-Belief Learningcited by 75
  3. 2024Building Machines that Learn and Think with Peoplecited by 73
  4. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  5. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 68
  6. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  7. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  8. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 58
  9. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 52
  10. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 49
  11. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 45
  12. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  13. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  14. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agentscited by 41
  15. 2021Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabicited by 40
  16. 2022Melting Pot 2.0cited by 40
  17. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  18. 2021On the Importance of Environments in Human-Robot Coordinationcited by 37
  19. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  20. 2024VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraftcited by 35
  21. 2025The Traitors: Deception and Trust in Multi-Agent Language Model Simulationscited by 34
  22. 2021A New Formalism, Method and Open Issues for Zero-Shot Coordinationcited by 33
  23. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  24. 2022K-level Reasoning for Zero-Shot Coordination in Hanabicited by 32
  25. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 32
  26. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  27. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 31
  28. 2022The Utility of Explainable AI in Ad Hoc Human-Machine Teamingcited by 30
  29. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  30. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 30
  31. 2021Deep Interactive Bayesian Reinforcement Learning via Meta-Learningcited by 29
  32. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  33. 2023A Review of Cooperation in Multi-agent Learningcited by 29
  34. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  35. 2021On the Critical Role of Conventions in Adaptive Human-AI Collaborationcited by 27
  36. 2022Performative Reinforcement Learningcited by 27
  37. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systemscited by 26
  38. 2021Evaluating the Robustness of Collaborative Agentscited by 25

50 most cited of 349 All 349 (JSON)