# Learning to Play No-Press Diplomacy with Best Response Policy Iteration

arXiv: https://arxiv.org/abs/2006.04635 · HTML: https://mlsanity.com/paper/2006.04635/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2020.

## Built on by (most cited first, 26 of 26)

- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning](https://mlsanity.com/paper/2210.05492/) (2022)
- [Welfare Diplomacy: Benchmarking Language Model Cooperation](https://mlsanity.com/paper/2310.08901/) (2023)
- [Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy](https://mlsanity.com/paper/2407.06813/) (2024)
- [No-Press Diplomacy from Scratch](https://mlsanity.com/paper/2110.02924/) (2021)
- [Student of Games: A unified learning algorithm for both perfect and imperfect information games](https://mlsanity.com/paper/2112.03178/) (2021)
- [Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers](https://mlsanity.com/paper/2106.09435/) (2021)
- [Towards Deeper Deep Reinforcement Learning with Spectral Normalization](https://mlsanity.com/paper/2106.01151/) (2021)
- [Solving Common-Payoff Games with Approximate Policy Iteration](https://mlsanity.com/paper/2101.04237/) (2021)
- [More Victories, Less Cooperation: Assessing Cicero's Diplomacy Play](https://mlsanity.com/paper/2406.04643/) (2024)
- [Choosing Well Your Opponents: How to Guide the Synthesis of Programmatic Strategies](https://mlsanity.com/paper/2307.04893/) (2023)
- [Normative Disagreement as a Challenge for Cooperative AI](https://mlsanity.com/paper/2111.13872/) (2021)
- [Dynamic Collaborative Multi-Agent Reinforcement Learning Communication for Autonomous Drone Reforestation](https://mlsanity.com/paper/2211.15414/) (2022)
- [Explaining Decisions of Agents in Mixed-Motive Games](https://mlsanity.com/paper/2407.15255/) (2024)
- [DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy](https://mlsanity.com/paper/2506.09655/) (2025)
- [Preference Communication in Multi-Objective Normal-Form Games](https://mlsanity.com/paper/2111.09191/) (2021)
- [Collaborative Auto-Curricula Multi-Agent Reinforcement Learning with Graph Neural Network Communication Layer for Open-ended Wildfire-Management Resource Distribution](https://mlsanity.com/paper/2204.11350/) (2022)
- [Game Theoretic Rating in N-player general-sum games with Equilibria](https://mlsanity.com/paper/2210.02205/) (2022)
- [Personalized Help for Optimizing Low-Skilled Users' Strategy](https://mlsanity.com/paper/2411.09109/) (2024)
- [Should I Trust You? Detecting Deception in Negotiations using Counterfactual RL](https://mlsanity.com/paper/2502.12436/) (2025)
- [Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments](https://mlsanity.com/paper/2209.10958/) (2022)
- [Understanding Iterative Combinatorial Auction Designs via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.19420/) (2024)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Cooperative Artificial Intelligence](https://mlsanity.com/paper/2202.09859/) (2022)
- [Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.12896/) (2022)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-the-art, and game theoretic equilibrium analysis shows that the new process yields consistent improvements.