# Collaborative Evolutionary Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.00976 · HTML: https://mlsanity.com/paper/1905.00976/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2019.

## Built on by (most cited first, 30 of 37)

- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Winner Takes It All: Training Performant RL Populations for Combinatorial Optimization](https://mlsanity.com/paper/2210.03475/) (2022)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Efficient Deep Reinforcement Learning Requires Regulating Overfitting](https://mlsanity.com/paper/2304.10466/) (2023)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation](https://mlsanity.com/paper/2210.17375/) (2022)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Neuroevolution-Enhanced Multi-Objective Optimization for Mixed-Precision Quantization](https://mlsanity.com/paper/2106.07611/) (2021)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Evolving Populations of Diverse RL Agents with MAP-Elites](https://mlsanity.com/paper/2303.12803/) (2023)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Games](https://mlsanity.com/paper/2209.10055/) (2022)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2404.08233/) (2024)
- [Evolutionary Policy Optimization](https://mlsanity.com/paper/2503.19037/) (2025)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Behavior-based Neuroevolutionary Training in Reinforcement Learning](https://mlsanity.com/paper/2105.07960/) (2021)
- [Cooperation and Competition: Flocking with Evolutionary Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.04696/) (2022)
- [Optimal Actor-Critic Policy with Optimized Training Datasets](https://mlsanity.com/paper/2108.06911/) (2021)
- [Evolving Constrained Reinforcement Learning Policy](https://mlsanity.com/paper/2304.09869/) (2023)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Evolutionary Reinforcement Learning via Cooperative Coevolution](https://mlsanity.com/paper/2404.14763/) (2024)
- [Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospects](https://mlsanity.com/paper/2407.08745/) (2024)
- [Surrogate-Assisted Evolutionary Reinforcement Learning Based on Autoencoder and Hyperbolic Neural Network](https://mlsanity.com/paper/2505.19423/) (2025)
- [The Price equation reveals a universal force-metric-bias law of algorithmic learning and natural selection](https://mlsanity.com/paper/2507.18549/) (2025)
- [Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies](https://mlsanity.com/paper/2508.03194/) (2025)
- [MOMA-AC: A preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learning](https://mlsanity.com/paper/2511.18181/) (2025)
- [REIN-2: Giving Birth to Prepared Reinforcement Learning Agents Using Reinforcement Learning Agents](https://mlsanity.com/paper/2110.05128/) (2021)

## Abstract (arXiv)

Deep reinforcement learning algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically struggle with achieving effective exploration and are extremely sensitive to the choice of hyperparameters. One reason is that most approaches use a noisy version of their operating policy to explore - thereby limiting the range of exploration. In this paper, we introduce Collaborative Evolutionary Reinforcement Learning (CERL), a scalable framework that comprises a portfolio of policies that simultaneously explore and exploit diverse regions of the solution space. A collection of learners - typically proven algorithms like TD3 - optimize over varying time-horizons leading to this diverse portfolio. All learners contribute to and use a shared replay buffer to achieve greater sample efficiency. Computational resources are dynamically distributed to favor the best learners as a form of online algorithm selection. Neuroevolution binds this entire process to generate a single emergent learner that exceeds the capabilities of any individual learner. Experiments in a range of continuous control benchmarks demonstrate that the emergent learner significantly outperforms its composite learners while remaining overall more sample-efficient - notably solving the Mujoco Humanoid benchmark where all of its composite learners (TD3) fail entirely in isolation.