# Evolutionary Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2303.04150 · HTML: https://mlsanity.com/paper/2303.04150/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 111 of its 261 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 30 of 111)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/1712.06567/) (2017)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents](https://mlsanity.com/paper/1712.06560/) (2017)

## Built on by (most cited first, 29 of 29)

- [Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents](https://mlsanity.com/paper/2311.11797/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models](https://mlsanity.com/paper/2402.01874/) (2024)
- [MetaDE: Evolving Differential Evolution by Differential Evolution](https://mlsanity.com/paper/2502.10470/) (2025)
- [Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities](https://mlsanity.com/paper/2308.13420/) (2023)
- [World Models with Hints of Large Language Models for Goal Achieving](https://mlsanity.com/paper/2406.07381/) (2024)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2404.08233/) (2024)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Evolutionary Reinforcement Learning via Cooperative Coevolution](https://mlsanity.com/paper/2404.14763/) (2024)
- [Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospects](https://mlsanity.com/paper/2407.08745/) (2024)
- [Near-Driven Autonomous Rover Navigation in Complex Environments: Extensions to Urban Search-and-Rescue and Industrial Inspection](https://mlsanity.com/paper/2504.17794/) (2025)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [A Denoising Diffusion-Based Evolutionary Algorithm Framework: Application to the Maximum Independent Set Problem](https://mlsanity.com/paper/2510.08627/) (2025)
- [Deep Reinforcement Learning-Assisted Automated Operator Portfolio for Constrained Multi-objective Optimization](https://mlsanity.com/paper/2603.16401/) (2026)
- [UCB-driven Utility Function Search for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2405.00410/) (2024)
- [Automating Creativity](https://mlsanity.com/paper/2405.06915/) (2024)
- [The Physics of Learning: From Autoencoders to Truly Autonomous Learning Machines](https://mlsanity.com/paper/2407.04700/) (2024)
- [Unlearning Works Better Than You Think: Local Reinforcement-Based Selection of Auxiliary Objectives](https://mlsanity.com/paper/2504.14418/) (2025)
- [Randomised Optimism via Competitive Co-Evolution for Matrix Games with Bandit Feedback](https://mlsanity.com/paper/2505.13562/) (2025)
- [PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2506.13741/) (2025)
- [Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2506.16795/) (2025)
- [Reinforcement learning in densely recurrent biological networks](https://mlsanity.com/paper/2508.09618/) (2025)
- [DyBBT: Dynamic Balance via Bandit-inspired Targeting for Dialog Policy with Cognitive Dual-Systems](https://mlsanity.com/paper/2509.19695/) (2025)
- [Harnessing Bounded-Support Evolution Strategies for Policy Refinement](https://mlsanity.com/paper/2511.09923/) (2025)
- [CosmoCore-Evo: Evolutionary Dream-Replay Reinforcement Learning for Adaptive Code Generation](https://mlsanity.com/paper/2512.21351/) (2025)
- [Robust Quadruped Locomotion via Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2604.07224/) (2026)
- [Learning to Orchestrate Evolutionary Search: Progression-Aware Deep Reinforcement Learning for Dynamic DE-CMA-ES Coordination in Optimization and Structural Model Updating](https://mlsanity.com/paper/2610.11546/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field.