# Emergent Coordination Through Competition

arXiv: https://arxiv.org/abs/1902.07151 · HTML: https://mlsanity.com/paper/1902.07151/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2019.

## Built on by (most cited first, 30 of 57)

- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [On the Critical Role of Conventions in Adaptive Human-AI Collaboration](https://mlsanity.com/paper/2104.02871/) (2021)
- [Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems](https://mlsanity.com/paper/2102.07659/) (2021)
- [Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization](https://mlsanity.com/paper/2204.02246/) (2022)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space](https://mlsanity.com/paper/2102.09812/) (2021)
- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations](https://mlsanity.com/paper/2110.04507/) (2021)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [NeuPL: Neural Population Learning](https://mlsanity.com/paper/2202.07415/) (2022)
- [Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2507.04724/) (2025)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL](https://mlsanity.com/paper/2106.15883/) (2021)
- [A Strategic Framework for Optimal Decisions in Football 1-vs-1 Shot-Taking Situations: An Integrated Approach of Machine Learning, Theory-Based Modeling, and Game Theory](https://mlsanity.com/paper/2307.14732/) (2023)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [How does AI play football? An analysis of RL and real-world football strategies](https://mlsanity.com/paper/2111.12340/) (2021)
- [MA-Dreamer: Coordination and communication through shared imagination](https://mlsanity.com/paper/2204.04687/) (2022)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Dealing with Non-Stationarity in MARL via Trust-Region Decomposition](https://mlsanity.com/paper/2102.10616/) (2021)
- [Two-stage training algorithm for AI robot soccer](https://mlsanity.com/paper/2104.05931/) (2021)
- [rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer](https://mlsanity.com/paper/2106.12895/) (2021)
- [Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Games](https://mlsanity.com/paper/2209.10055/) (2022)

## Abstract (arXiv)

We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.