# Evolutionary Reinforcement Learning via Cooperative Coevolution

arXiv: https://arxiv.org/abs/2404.14763 · HTML: https://mlsanity.com/paper/2404.14763/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Evolution-Guided Policy Gradient in Reinforcement Learning](https://mlsanity.com/paper/1805.07917/) (2018)
- [CEM-RL: Combining evolutionary and gradient-based methods for policy search](https://mlsanity.com/paper/1810.01222/) (2018)
- [Collaborative Evolutionary Reinforcement Learning](https://mlsanity.com/paper/1905.00976/) (2019)
- [Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges](https://mlsanity.com/paper/2006.05415/) (2020)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Proximal Distilled Evolutionary Reinforcement Learning](https://mlsanity.com/paper/1906.09807/) (2019)
- [ES Is More Than Just a Traditional Finite-Difference Approximator](https://mlsanity.com/paper/1712.06568/) (2017)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients](https://mlsanity.com/paper/1712.06563/) (2017)
- [Reducing Idleness in Financial Cloud Services via Multi-objective Evolutionary Reinforcement Learning based Load Balancer](https://mlsanity.com/paper/2305.03463/) (2023)
- [A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2201.00129/) (2022)
- [Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search](https://mlsanity.com/paper/2009.03603/) (2020)
- [High-dimensional Black-box Optimization via Divide and Approximate Conquer](https://mlsanity.com/paper/1603.03518/) (2016)
- [Policy Optimization by Genetic Distillation](https://mlsanity.com/paper/1711.01012/) (2017)
- [Algorithm Portfolios for Noisy Optimization](https://mlsanity.com/paper/1511.01277/) (2015)

## Built on by (most cited first, 1 of 1)

- [Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training](https://mlsanity.com/paper/2608.02391/) (2026)

## Abstract (arXiv)

Recently, evolutionary reinforcement learning has obtained much attention in various domains. Maintaining a population of actors, evolutionary reinforcement learning utilises the collected experiences to improve the behaviour policy through efficient exploration. However, the poor scalability of genetic operators limits the efficiency of optimising high-dimensional neural networks.To address this issue, this paper proposes a novel cooperative coevolutionary reinforcement learning (CoERL) algorithm. Inspired by cooperative coevolution, CoERL periodically and adaptively decomposes the policy optimisation problem into multiple subproblems and evolves a population of neural networks for each of the subproblems. Instead of using genetic operators, CoERL directly searches for partial gradients to update the policy. Updating policy with partial gradients maintains consistency between the behaviour spaces of parents and offspring across generations.The experiences collected by the population are then used to improve the entire policy, which enhances the sampling efficiency.Experiments on six benchmark locomotion tasks demonstrate that CoERL outperforms seven state-of-the-art algorithms and baselines.Ablation study verifies the unique contribution of CoERL's core ingredients.