# Emergent Complexity via Multi-Agent Competition

arXiv: https://arxiv.org/abs/1710.03748 · HTML: https://mlsanity.com/paper/1710.03748/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2017.

## Built on by (most cited first, 30 of 120)

- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning](https://mlsanity.com/paper/2105.00579/) (2021)
- [Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi](https://mlsanity.com/paper/2107.07630/) (2021)
- [PMP: Learning to Physically Interact with Environments using Part-wise Motion Priors](https://mlsanity.com/paper/2305.03249/) (2023)
- [XDO: A Double Oracle Algorithm for Extensive-Form Games](https://mlsanity.com/paper/2103.06426/) (2021)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets](https://mlsanity.com/paper/2210.04688/) (2022)
- [On the Limitations of Elo: Real-World Games, are Transitive, not Additive](https://mlsanity.com/paper/2206.12301/) (2022)
- [The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process](https://mlsanity.com/paper/2506.01080/) (2025)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space](https://mlsanity.com/paper/2102.09812/) (2021)
- [A social path to human-like artificial intelligence](https://mlsanity.com/paper/2405.15815/) (2024)
- [PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning](https://mlsanity.com/paper/2202.03609/) (2022)
- [Collaborative Visual Navigation](https://mlsanity.com/paper/2107.01151/) (2021)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Zero-Shot Reinforcement Learning via Function Encoders](https://mlsanity.com/paper/2401.17173/) (2024)
- [ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation](https://mlsanity.com/paper/2109.06795/) (2021)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models](https://mlsanity.com/paper/2406.04274/) (2024)
- [Eurekaverse: Environment Curriculum Generation via Large Language Models](https://mlsanity.com/paper/2411.01775/) (2024)
- [It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation](https://mlsanity.com/paper/2202.10608/) (2022)
- [Toward Optimal LLM Alignments Using Two-Player Games](https://mlsanity.com/paper/2406.10977/) (2024)
- [A Review of Nine Physics Engines for Reinforcement Learning Research](https://mlsanity.com/paper/2407.08590/) (2024)
- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)

## Abstract (arXiv)

Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX