# Diverse Conventions for Human-AI Collaboration

arXiv: https://arxiv.org/abs/2310.15414 · HTML: https://mlsanity.com/paper/2310.15414/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi](https://mlsanity.com/paper/2107.07630/) (2021)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [Too many cooks: Bayesian inference for coordinating multi-agent collaboration](https://mlsanity.com/paper/2003.11778/) (2020)
- [On the Critical Role of Conventions in Adaptive Human-AI Collaboration](https://mlsanity.com/paper/2104.02871/) (2021)
- [Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations](https://mlsanity.com/paper/2010.15896/) (2020)
- [Conditional Imitation Learning for Multi-Agent Games](https://mlsanity.com/paper/2201.01448/) (2022)
- [Emergent Prosociality in Multi-Agent Games Through Gifting](https://mlsanity.com/paper/2105.06593/) (2021)
- [Adaptable Agent Populations via a Generative Model of Policies](https://mlsanity.com/paper/2107.07506/) (2021)
- [PantheonRL: A MARL Library for Dynamic Training Interactions](https://mlsanity.com/paper/2112.07013/) (2021)

## Built on by (most cited first, 15 of 15)

- [Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence](https://mlsanity.com/paper/2404.00029/) (2024)
- [Learning to Cooperate with Humans using Generative Agents](https://mlsanity.com/paper/2411.13934/) (2024)
- [ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork](https://mlsanity.com/paper/2505.23686/) (2025)
- [Adaptively Coordinating with Novel Partners via Learned Latent Strategies](https://mlsanity.com/paper/2511.12754/) (2025)
- [Improving Human-AI Coordination through Online Adversarial Training and Generative Models](https://mlsanity.com/paper/2504.15457/) (2025)
- [Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractions](https://mlsanity.com/paper/2505.04579/) (2025)
- [Robust and Diverse Multi-Agent Learning via Rational Policy Gradient](https://mlsanity.com/paper/2511.09535/) (2025)
- [Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork](https://mlsanity.com/paper/2605.24423/) (2026)
- [Automatic Curriculum Design for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2503.07275/) (2025)
- [Efficient Generation of Diverse Cooperative Agents with World Models](https://mlsanity.com/paper/2506.07450/) (2025)
- [Partner Capability Estimation for Task-Agnostic Adaptation in Ad-Hoc Teamwork](https://mlsanity.com/paper/2607.27177/) (2026)
- [JaxAHT: A JAX-Based Library for Ad Hoc Teamwork](https://mlsanity.com/paper/2609.13716/) (2026)
- [Moving Out: Physically-grounded Human-AI Collaboration](https://mlsanity.com/paper/2507.18623/) (2025)
- [Beyond Partner Diversity: An Influence-Based Team Steering Framework for Zero-Shot Human-Machine Teaming](https://mlsanity.com/paper/2605.15400/) (2026)
- [PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination](https://mlsanity.com/paper/2610.15969/) (2026)

## Abstract (arXiv)

Conventions are crucial for strong performance in cooperative multi-agent games, because they allow players to coordinate on a shared strategy without explicit communication. Unfortunately, standard multi-agent reinforcement learning techniques, such as self-play, converge to conventions that are arbitrary and non-diverse, leading to poor generalization when interacting with new partners. In this work, we present a technique for generating diverse conventions by (1) maximizing their rewards during self-play, while (2) minimizing their rewards when playing with previously discovered conventions (cross-play), stimulating conventions to be semantically different. To ensure that learned policies act in good faith despite the adversarial optimization of cross-play, we introduce mixed-play, where an initial state is randomly generated by sampling self-play and cross-play transitions and the player learns to maximize the self-play reward from this initial state. We analyze the benefits of our technique on various multi-agent collaborative games, including Overcooked, and find that our technique can adapt to the conventions of humans, surpassing human-level performance when paired with real users.