# Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

arXiv: https://arxiv.org/abs/1906.07315 · HTML: https://mlsanity.com/paper/1906.07315/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 14 of 14)

- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [XDO: A Double Oracle Algorithm for Extensive-Form Games](https://mlsanity.com/paper/2103.06426/) (2021)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation](https://mlsanity.com/paper/2210.17375/) (2022)
- [Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2404.08233/) (2024)
- [Improving Social Welfare While Preserving Autonomy via a Pareto Mediator](https://mlsanity.com/paper/2106.03927/) (2021)
- [Minimizing Communication while Maximizing Performance in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.08482/) (2021)
- [Cooperation and Competition: Flocking with Evolutionary Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.04696/) (2022)
- [Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07729/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Game Theory and Multi-Agent Reinforcement Learning : From Nash Equilibria to Evolutionary Dynamics](https://mlsanity.com/paper/2412.20523/) (2024)
- [Offsetting Unequal Competition through RL-assisted Incentive Schemes](https://mlsanity.com/paper/2201.01450/) (2022)
- [BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization](https://mlsanity.com/paper/2308.01207/) (2023)

## Abstract (arXiv)

Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Furthermore, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We introduce Multiagent Evolutionary Reinforcement Learning (MERL), a split-level training platform that handles the two objectives separately through two optimization processes. An evolutionary algorithm maximizes the sparse team-based objective through neuroevolution on a population of teams. Concurrently, a gradient-based optimizer trains policies to only maximize the dense agent-specific rewards. The gradient-based policies are periodically added to the evolutionary population as a way of information transfer between the two optimization processes. This enables the evolutionary algorithm to use skills learned via the agent-specific rewards toward optimizing the global objective. Results demonstrate that MERL significantly outperforms state-of-the-art methods, such as MADDPG, on a number of difficult coordination benchmarks.