# Evolution-Guided Policy Gradient in Reinforcement Learning

arXiv: https://arxiv.org/abs/1805.07917 · HTML: https://mlsanity.com/paper/1805.07917/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 30 of 83)

- [Symbolic Regression via Neural-Guided Genetic Programming Population Seeding](https://mlsanity.com/paper/2111.00053/) (2021)
- [For SALE: State-Action Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2306.02451/) (2023)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [Winner Takes It All: Training Performant RL Populations for Combinatorial Optimization](https://mlsanity.com/paper/2210.03475/) (2022)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model](https://mlsanity.com/paper/2105.15089/) (2021)
- [CFlowNets: Continuous Control with Generative Flow Networks](https://mlsanity.com/paper/2303.02430/) (2023)
- [PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference](https://mlsanity.com/paper/2410.21966/) (2024)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning](https://mlsanity.com/paper/2205.00293/) (2022)
- [B2Opt: Learning to Optimize Black-box Optimization with Little Budget](https://mlsanity.com/paper/2304.11787/) (2023)
- [ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation](https://mlsanity.com/paper/2210.17375/) (2022)
- [Sparse Reward Exploration via Novelty Search and Emitters](https://mlsanity.com/paper/2102.03140/) (2021)
- [Evolutionary Deep Reinforcement Learning for Dynamic Slice Management in O-RAN](https://mlsanity.com/paper/2208.14394/) (2022)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation](https://mlsanity.com/paper/2503.24322/) (2025)
- [ReDit: Reward Dithering for Improved LLM Policy Optimization](https://mlsanity.com/paper/2506.18631/) (2025)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidance](https://mlsanity.com/paper/2212.01939/) (2022)
- [Making Reinforcement Learning Work on Swimmer](https://mlsanity.com/paper/2208.07587/) (2022)
- [Learning to Transfer for Evolutionary Multitasking](https://mlsanity.com/paper/2406.14359/) (2024)
- [Evolving Populations of Diverse RL Agents with MAP-Elites](https://mlsanity.com/paper/2303.12803/) (2023)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [Evolutionary System Prompt Learning for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2602.14697/) (2026)
- [Building Human-like Communicative Intelligence: A Grounded Perspective](https://mlsanity.com/paper/2201.02734/) (2022)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically suffer from three core difficulties: temporal credit assignment with sparse rewards, lack of effective exploration, and brittle convergence properties that are extremely sensitive to hyperparameters. Collectively, these challenges severely limit the applicability of these approaches to real-world problems. Evolutionary Algorithms (EAs), a class of black box optimization techniques inspired by natural evolution, are well suited to address each of these three challenges. However, EAs typically suffer from high sample complexity and struggle to solve problems that require optimization of a large number of parameters. In this paper, we introduce Evolutionary Reinforcement Learning (ERL), a hybrid algorithm that leverages the population of an EA to provide diversified data to train an RL agent, and reinserts the RL agent into the EA population periodically to inject gradient information into the EA. ERL inherits EA's ability of temporal credit assignment with a fitness metric, effective exploration with a diverse set of policies, and stability of a population-based approach and complements it with off-policy DRL's ability to leverage gradients for higher sample efficiency and faster learning. Experiments in a range of challenging continuous control benchmarks demonstrate that ERL significantly outperforms prior DRL and EA methods.