# Evolutionary Strategy Guided Reinforcement Learning via MultiBuffer Communication

arXiv: https://arxiv.org/abs/2306.11535 · HTML: https://mlsanity.com/paper/2306.11535/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents](https://mlsanity.com/paper/1712.06560/) (2017)
- [Evolution-Guided Policy Gradient in Reinforcement Learning](https://mlsanity.com/paper/1805.07917/) (2018)
- [CEM-RL: Combining evolutionary and gradient-based methods for policy search](https://mlsanity.com/paper/1810.01222/) (2018)
- [Proximal Distilled Evolutionary Reinforcement Learning](https://mlsanity.com/paper/1906.09807/) (2019)
- [ES Is More Than Just a Traditional Finite-Difference Approximator](https://mlsanity.com/paper/1712.06568/) (2017)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Population-Guided Parallel Policy Search for Reinforcement Learning](https://mlsanity.com/paper/2001.02907/) (2020)
- [An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search](https://mlsanity.com/paper/2012.05417/) (2020)
- [Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments](https://mlsanity.com/paper/2205.07015/) (2022)
- [Cooperative Heterogeneous Deep Reinforcement Learning](https://mlsanity.com/paper/2011.00791/) (2020)
- [Making Reinforcement Learning Work on Swimmer](https://mlsanity.com/paper/2208.07587/) (2022)
- [FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control](https://mlsanity.com/paper/1907.00526/) (2019)
- [Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Games](https://mlsanity.com/paper/2209.10055/) (2022)
- [Evolutionary Deep Reinforcement Learning Using Elite Buffer: A Novel Approach Towards DRL Combined with EA in Continuous Control Tasks](https://mlsanity.com/paper/2209.08480/) (2022)

## Built on by (most cited first, 2 of 2)

- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning](https://mlsanity.com/paper/2408.09493/) (2024)

## Abstract (arXiv)

Evolutionary Algorithms and Deep Reinforcement Learning have both successfully solved control problems across a variety of domains. Recently, algorithms have been proposed which combine these two methods, aiming to leverage the strengths and mitigate the weaknesses of both approaches. In this paper we introduce a new Evolutionary Reinforcement Learning model which combines a particular family of Evolutionary algorithm called Evolutionary Strategies with the off-policy Deep Reinforcement Learning algorithm TD3. The framework utilises a multi-buffer system instead of using a single shared replay buffer. The multi-buffer system allows for the Evolutionary Strategy to search freely in the search space of policies, without running the risk of overpopulating the replay buffer with poorly performing trajectories which limit the number of desirable policy behaviour examples thus negatively impacting the potential of the Deep Reinforcement Learning within the shared framework. The proposed algorithm is demonstrated to perform competitively with current Evolutionary Reinforcement Learning algorithms on MuJoCo control tasks, outperforming the well known state-of-the-art CEM-RL on 3 of the 4 environments tested.