# Towards Learning Multi-agent Negotiations via Self-Play

arXiv: https://arxiv.org/abs/2001.10208 · HTML: https://mlsanity.com/paper/2001.10208/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2020.

## Built on by (most cited first, 17 of 17)

- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Models](https://mlsanity.com/paper/2104.10558/) (2021)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)
- [Nav-Q: Quantum Deep Reinforcement Learning for Collision-Free Navigation of Self-Driving Cars](https://mlsanity.com/paper/2311.12875/) (2023)
- [Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios](https://mlsanity.com/paper/2103.12070/) (2021)
- [Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2110.08642/) (2021)
- [Hybrid Deep Reinforcement Learning and Planning for Safe and Comfortable Automated Driving](https://mlsanity.com/paper/2301.00650/) (2022)
- [Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2209.10003/) (2022)
- [Intent-Aware Autonomous Driving: A Case Study on Highway Merging Scenarios](https://mlsanity.com/paper/2309.13206/) (2023)
- [Pictura: Perspective-View Self-Play at Scale for Driving](https://mlsanity.com/paper/2607.26005/) (2026)
- [Rationale-aware Autonomous Driving Policy utilizing Safety Force Field implemented on CARLA Simulator](https://mlsanity.com/paper/2211.10237/) (2022)
- [Truthful Self-Play](https://mlsanity.com/paper/2106.03007/) (2021)
- [Cognitive Insights and Stable Coalition Matching for Fostering Multi-Agent Cooperation](https://mlsanity.com/paper/2405.18044/) (2024)

## Abstract (arXiv)

Making sophisticated, robust, and safe sequential decisions is at the heart of intelligent systems. This is especially critical for planning in complex multi-agent environments, where agents need to anticipate other agents' intentions and possible future actions. Traditional methods formulate the problem as a Markov Decision Process, but the solutions often rely on various assumptions and become brittle when presented with corner cases. In contrast, deep reinforcement learning (Deep RL) has been very effective at finding policies by simultaneously exploring, interacting, and learning from environments. Leveraging the powerful Deep RL paradigm, we demonstrate that an iterative procedure of self-play can create progressively more diverse environments, leading to the learning of sophisticated and robust multi-agent policies. We demonstrate this in a challenging multi-agent simulation of merging traffic, where agents must interact and negotiate with others in order to successfully merge on or off the road. While the environment starts off simple, we increase its complexity by iteratively adding an increasingly diverse set of agents to the agent "zoo" as training progresses. Qualitatively, we find that through self-play, our policies automatically learn interesting behaviors such as defensive driving, overtaking, yielding, and the use of signal lights to communicate intentions to other agents. In addition, quantitatively, we show a dramatic improvement of the success rate of merging maneuvers from 63% to over 98%.