Genetic Drift Regularization: on preventing Actor Injection from breaking Evolution Strategies
On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization
Abstract (arXiv)
Evolutionary Algorithms (EA) have been successfully used for the optimization of neural networks for policy search, but they still remain sample inefficient and underperforming in some cases compared to gradient-based reinforcement learning (RL). Various methods combine the two approaches, many of them training a RL algorithm on data from EA evaluations and injecting the RL actor into the EA population. However, when using Evolution Strategies (ES) as the EA, the RL actor can drift genetically far from the the ES distribution and injection can cause a collapse of the ES performance. Here, we highlight the phenomenon of genetic drift where the actor genome and the ES population distribution progressively drift apart, leading to injection having a negative impact on the ES. We introduce Genetic Drift Regularization (GDR), a simple regularization method in the actor training loss that prevents the actor genome from drifting away from the ES. We show that GDR can improve ES convergence on problems where RL learns well, but also helps RL training on other tasks, , fixes the injection issues better than previous controlled injection methods.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 39 references that are in the map. First posted on arXiv 7 May 2024.
Builds on
16 of 39 references.
References · 16
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
- 2018Evolution-Guided Policy Gradient in Reinforcement Learningcited by 83
- 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 58
- 2022evosax: JAX-based Evolution Strategiescited by 41
- 2019Proximal Distilled Evolutionary Reinforcement Learningcited by 27
- 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 23
- 2022ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representationcited by 14
4 more
- 2022A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learningcited by 13
- 2023Understanding the Synergies between Quality-Diversity and Deep Reinforcement Learningcited by 6
- 2023Rethinking Population-assisted Off-policy Reinforcement Learningcited by 5
- 2023Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainabilitycited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning
- 2026Continual Reinforcement Learning with Neuroevolution
- 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms
- 2025Synergizing Reinforcement Learning and Genetic Algorithms for Neural Combinatorial Optimization
- 2024Deep Reinforcement Learning for Dynamic Algorithm Selection: A Proof-of-Principle Study on Differential Evolution
- 2026RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms
- 2024Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks
- 2024Evolutionary Reinforcement Learning: A Systematic Review and Future Directions