# Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2003.10423 · HTML: https://mlsanity.com/paper/2003.10423/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 30 of 48)

- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems](https://mlsanity.com/paper/2111.04613/) (2021)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft](https://mlsanity.com/paper/2412.05255/) (2024)
- [Learning Efficient Multi-Agent Cooperative Visual Exploration](https://mlsanity.com/paper/2110.05734/) (2021)
- [Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition](https://mlsanity.com/paper/2211.12712/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Multi-Agent Policy Transfer via Task Relationship Modeling](https://mlsanity.com/paper/2203.04482/) (2022)
- [On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2111.05992/) (2021)
- [Sable: a Performant, Efficient and Scalable Sequence Model for MARL](https://mlsanity.com/paper/2410.01706/) (2024)
- [ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.16186/) (2023)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation](https://mlsanity.com/paper/2503.21200/) (2025)
- [Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.06042/) (2021)
- [Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.12988/) (2021)
- [Breaking the Curse of Dimensionality in Multiagent State Space: A Unified Agent Permutation Framework](https://mlsanity.com/paper/2203.05285/) (2022)
- [Learning Continuous Control Policies for Information-Theoretic Active Perception](https://mlsanity.com/paper/2209.12427/) (2022)
- [Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03429/) (2023)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation](https://mlsanity.com/paper/2106.05907/) (2021)
- [Learning Progress Driven Multi-Agent Curriculum](https://mlsanity.com/paper/2205.10016/) (2022)
- [S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2206.11054/) (2022)
- [Policy Learning for Active Target Tracking over Continuous SE(3) Trajectories](https://mlsanity.com/paper/2212.01498/) (2022)
- [Iteratively Learn Diverse Strategies with State Distance Information](https://mlsanity.com/paper/2310.14509/) (2023)
- [Inverse Attention Agents for Multi-Agent Systems](https://mlsanity.com/paper/2410.21794/) (2024)
- [Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning](https://mlsanity.com/paper/2310.04796/) (2023)
- [SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.01839/) (2024)
- [SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks](https://mlsanity.com/paper/2503.11726/) (2025)

## Abstract (arXiv)

In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.