# An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search

arXiv: https://arxiv.org/abs/2012.05417 · HTML: https://mlsanity.com/paper/2012.05417/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 12 of 12)

- [Learning to run a Power Network Challenge: a Retrospective Analysis](https://mlsanity.com/paper/2103.03104/) (2021)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [PowRL: A Reinforcement Learning Framework for Robust Management of Power Networks](https://mlsanity.com/paper/2212.02397/) (2022)
- [Reinforcement learning for Energies of the future and carbon neutrality: a Challenge Design](https://mlsanity.com/paper/2207.10330/) (2022)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [A Simple Decentralized Cross-Entropy Method](https://mlsanity.com/paper/2212.08235/) (2022)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [DRL-ISP: Multi-Objective Camera ISP with Deep Reinforcement Learning](https://mlsanity.com/paper/2207.03081/) (2022)
- [Evolution Guided Generative Flow Networks](https://mlsanity.com/paper/2402.02186/) (2024)
- [Evolutionary Strategy Guided Reinforcement Learning via MultiBuffer Communication](https://mlsanity.com/paper/2306.11535/) (2023)
- [Supplementing Gradient-Based Reinforcement Learning with Simple Evolutionary Ideas](https://mlsanity.com/paper/2305.07571/) (2023)

## Abstract (arXiv)

Deep reinforcement learning (DRL) algorithms and evolution strategies (ES) have been applied to various tasks, showing excellent performances. These have the opposite properties, with DRL having good sample efficiency and poor stability, while ES being vice versa. Recently, there have been attempts to combine these algorithms, but these methods fully rely on synchronous update scheme, making it not ideal to maximize the benefits of the parallelism in ES. To solve this challenge, asynchronous update scheme was introduced, which is capable of good time-efficiency and diverse policy exploration. In this paper, we introduce an Asynchronous Evolution Strategy-Reinforcement Learning (AES-RL) that maximizes the parallel efficiency of ES and integrates it with policy gradient methods. Specifically, we propose 1) a novel framework to merge ES and DRL asynchronously and 2) various asynchronous update methods that can take all advantages of asynchronism, ES, and DRL, which are exploration and time efficiency, stability, and sample efficiency, respectively. The proposed framework and update methods are evaluated in continuous control benchmark work, showing superior performance as well as time efficiency compared to the previous methods.