mlsanityData to 7 Oct 2026

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

arXiv 2112.11701 ↗First posted 22 Dec 2021cs.AIcited by 69

Not on the map. Why?

Abstract (arXiv)

We study the problem of training a Reinforcement Learning (RL) agent that is collaborative with humans without using any human data. Although such agents can be obtained through self-play training, they can suffer significantly from distributional shift when paired with unencountered partners, such as humans. To mitigate this distributional shift, we propose Maximum Entropy Population-based training (MEP). In MEP, agents in the population are trained with our derived Population Entropy bonus to promote both pairwise diversity between agents and individual diversity of agents themselves, and a common best agent is trained by paring with agents in this diversified population via prioritized sampling. The prioritization is dynamically adjusted based on the training progress. We demonstrate the effectiveness of our method MEP, with comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory Diversity (TrajeDi), and Fictitious Co-Play (FCP) in the Overcooked game environment, with partners being human proxy models and real humans. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 38 of its 60 references that are in the map. First posted on arXiv 22 Dec 2021.

Builds on

38 of 60 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2018Learning Dexterous In-Hand Manipulationcited by 643
26 more
  1. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  2. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  3. 2017Population Based Training of Neural Networkscited by 351
  4. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  5. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  6. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  7. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  8. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  9. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  10. 2021Collaborating with Humans without Human Datacited by 139
  11. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  12. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  13. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  14. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  15. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
  16. 2017Maintaining cooperation in complex social dilemmas using deep reinforcement learningcited by 48
  17. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  18. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  19. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  20. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  21. 2019Theory of Minds: Understanding Behavior in Groups Through Inverse Planningcited by 30
  22. 2021Evaluating the Robustness of Collaborative Agentscited by 25
  23. 2020TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Gamecited by 21
  24. 2021Mutual Information State Intrinsic Controlcited by 18
  25. 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
  26. 2020Adversarially Guided Self-Play for Adopting Social Conventionscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 13; 2025: 26; 2026: 12.

69 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  2. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  3. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 65
  4. 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
  5. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  6. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  7. 2023A Review of Cooperation in Multi-agent Learningcited by 27
  8. 2023PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordinationcited by 25
  9. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  10. 2024Learning to Cooperate with Humans using Generative Agentscited by 23
  11. 2025Virtual Agent Economiescited by 17
  12. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
38 more
  1. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
  2. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
  3. 2022Heterogeneous Multi-agent Zero-Shot Coordination by Coevolutioncited by 12
  4. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  5. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  6. 2025MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learningcited by 8
  7. 2023A Hierarchical Approach to Population Training for Human-AI Collaborationcited by 7
  8. 2023Efficient Human-AI Coordination via Preparatory Language-based Conventioncited by 7
  9. 2023Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generationcited by 6
  10. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  11. 2025ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamworkcited by 6
  12. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  13. 2023Mixture of personality improved Spiking actor network for efficient multi-agent cooperationcited by 4
  14. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  15. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  16. 2024BCR-DRL: Behavior- and Context-aware Reward for Deep Reinforcement Learning in Human-AI Coordinationcited by 3
  17. 2024Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactionscited by 3
  18. 2025Improving Human-AI Coordination through Online Adversarial Training and Generative Modelscited by 3
  19. 2025The Yokai Learning Environment: Tracking Beliefs Over Space and Timecited by 3
  20. 2025Adaptively Coordinating with Novel Partners via Learned Latent Strategiescited by 3
  21. 2024Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptationcited by 2
  22. 2024KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordinationcited by 2
  23. 2025Who is Helping Whom? Analyzing Inter-dependencies to Evaluate Cooperation in Human-AI Teamingcited by 2
  24. 2025Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractionscited by 2
  25. 2025Unsupervised Partner Design Enables Robust Ad-hoc Teamworkcited by 2
  26. 2025CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planningcited by 2
  27. 2026GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learningcited by 2
  28. 2026Training Generalizable Collaborative Agents via Strategic Risk Aversioncited by 2
  29. 2026Shaping Zero-Shot Coordination via State Blockingcited by 2
  30. 2026Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamworkcited by 2
  31. 2026AI for Games in the Foundation Model Eracited by 2
  32. 2024Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Gamescited by 1
  33. 2024HOLA-Drone: Hypergraphic Open-ended Learning for Zero-Shot Multi-Drone Cooperative Pursuitcited by 1
  34. 2025Automatic Curriculum Design for Zero-Shot Human-AI Coordinationcited by 1
  35. 2025Enhancing Sample Efficiency in Multi-Agent RL with Uncertainty Quantification and Selective Explorationcited by 1
  36. 2025Efficient Generation of Diverse Cooperative Agents with World Modelscited by 1
  37. 2025Modeling Latent Partner Strategies for Adaptive Zero-Shot Human-Agent Collaborationcited by 1
  38. 2025Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewardscited by 1

50 most cited of 69 All 69 (JSON)