# Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments

arXiv: https://arxiv.org/abs/2209.10958 · HTML: https://mlsanity.com/paper/2209.10958/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 98 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 30 of 63)

- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [A multi-agent reinforcement learning model of common-pool resource appropriation](https://mlsanity.com/paper/1707.06600/) (2017)
- [Real World Games Look Like Spinning Tops](https://mlsanity.com/paper/2004.09468/) (2020)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research](https://mlsanity.com/paper/1903.00742/) (2019)
- [From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization](https://mlsanity.com/paper/2002.08456/) (2020)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)
- [Learning to Incentivize Other Learning Agents](https://mlsanity.com/paper/2006.06051/) (2020)
- [Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint](https://mlsanity.com/paper/2106.03787/) (2021)
- [A Generalized Training Approach for Multiagent Learning](https://mlsanity.com/paper/1909.12823/) (2019)
- [α-Rank: Multi-Agent Evaluation by Evolution](https://mlsanity.com/paper/1903.01373/) (2019)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Biases for Emergent Communication in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1912.05676/) (2019)
- [Re-evaluating Evaluation](https://mlsanity.com/paper/1806.02643/) (2018)
- [Human-Level Performance in No-Press Diplomacy via Equilibrium Search](https://mlsanity.com/paper/2010.02923/) (2020)
- [EigenGame: PCA as a Nash Equilibrium](https://mlsanity.com/paper/2010.00554/) (2020)
- [No Press Diplomacy: Modeling Multi-Agent Gameplay](https://mlsanity.com/paper/1909.02128/) (2019)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Actor-Critic Policy Optimization in Partially Observable Multiagent Environments](https://mlsanity.com/paper/1810.09026/) (2018)

## Built on by (most cited first, 2 of 2)

- [Online Learning and Solving Infinite Games with an ERM Oracle](https://mlsanity.com/paper/2307.01689/) (2023)
- [The Hidden Game Problem](https://mlsanity.com/paper/2510.03845/) (2025)

## Abstract (arXiv)

The Game Theory & Multi-Agent team at DeepMind studies several aspects of multi-agent learning ranging from computing approximations to fundamental concepts in game theory to simulating social dilemmas in rich spatial environments and training 3-d humanoids in difficult team coordination tasks. A signature aim of our group is to use the resources and expertise made available to us at DeepMind in deep reinforcement learning to explore multi-agent systems in complex environments and use these benchmarks to advance our understanding. Here, we summarise the recent work of our team and present a taxonomy that we feel highlights many important open challenges in multi-agent research.