# A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.00832 · HTML: https://mlsanity.com/paper/1711.00832/

## Key facts

- Cited by 268 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2017.

## Built on by (most cited first, 30 of 268)

- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06135/) (2021)
- [A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games](https://mlsanity.com/paper/2206.05825/) (2022)
- [Evaluating Language Model Agency through Negotiations](https://mlsanity.com/paper/2401.04536/) (2024)
- [Learning Multi-Agent Communication from Graph Modeling Perspective](https://mlsanity.com/paper/2405.08550/) (2024)
- [Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System](https://mlsanity.com/paper/2410.08115/) (2024)
- [Melting Pot 2.0](https://mlsanity.com/paper/2211.13746/) (2022)
- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models](https://mlsanity.com/paper/2506.07468/) (2025)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Independent Natural Policy Gradient Always Converges in Markov Potential Games](https://mlsanity.com/paper/2110.10614/) (2021)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [XDO: A Double Oracle Algorithm for Extensive-Form Games](https://mlsanity.com/paper/2103.06426/) (2021)
- [Gradient play in stochastic games: stationary points, convergence, and sample complexity](https://mlsanity.com/paper/2106.00198/) (2021)
- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [K-level Reasoning for Zero-Shot Coordination in Hanabi](https://mlsanity.com/paper/2207.07166/) (2022)
- [No-Press Diplomacy from Scratch](https://mlsanity.com/paper/2110.02924/) (2021)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [Cooperative Open-ended Learning Framework for Zero-shot Coordination](https://mlsanity.com/paper/2302.04831/) (2023)
- [Adversarial Policies Beat Superhuman Go AIs](https://mlsanity.com/paper/2211.00241/) (2022)
- [Steering Language Models with Game-Theoretic Solvers](https://mlsanity.com/paper/2402.01704/) (2024)
- [ASP: Learn a Universal Neural Solver!](https://mlsanity.com/paper/2303.00466/) (2023)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)

## Abstract (arXiv)

To achieve general intelligence, agents must learn how to interact with others in a shared environment: this is the challenge of multiagent reinforcement learning (MARL). The simplest form is independent reinforcement learning (InRL), where each agent treats its experience as part of its (non-stationary) environment. In this paper, we first observe that policies learned using InRL can overfit to the other agents' policies during training, failing to sufficiently generalize during execution. We introduce a new metric, joint-policy correlation, to quantify this effect. We describe an algorithm for general MARL, based on approximate best responses to mixtures of policies generated using deep reinforcement learning, and empirical game-theoretic analysis to compute meta-strategies for policy selection. The algorithm generalizes previous ones such as InRL, iterated best response, double oracle, and fictitious play. Then, we present a scalable implementation which reduces the memory requirement using decoupled meta-solvers. Finally, we demonstrate the generality of the resulting policies in two partially observable settings: gridworld coordination games and poker.