# Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

arXiv: https://arxiv.org/abs/2110.01411 · HTML: https://mlsanity.com/paper/2110.01411/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 131 of its 313 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 30 of 131)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)

## Built on by (most cited first, 28 of 28)

- [Open-environment Machine Learning](https://mlsanity.com/paper/2206.00423/) (2022)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Games for Artificial Intelligence Research: A Review and Perspectives](https://mlsanity.com/paper/2304.13269/) (2023)
- [Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving](https://mlsanity.com/paper/2501.08096/) (2025)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Guided Reinforcement Learning for Omnidirectional 3D Jumping in Quadruped Robots](https://mlsanity.com/paper/2507.16481/) (2025)
- [Pearl: Parallel Evolutionary and Reinforcement Learning Library](https://mlsanity.com/paper/2201.09568/) (2022)
- [UnifiedRL: A Reinforcement Learning Algorithm Tailored for Multi-Task Fusion in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.17589/) (2024)
- [Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning](https://mlsanity.com/paper/2501.13883/) (2025)
- [SMC-ES: Automated synthesis of formally verified control policies](https://mlsanity.com/paper/2607.15003/) (2026)
- [Efficient Reinforcement Learning for Jumping Monopods](https://mlsanity.com/paper/2309.07038/) (2023)
- [Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospects](https://mlsanity.com/paper/2407.08745/) (2024)
- [The Digital Ecosystem of Beliefs: does evolution favour AI over humans?](https://mlsanity.com/paper/2412.14500/) (2024)
- [Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning](https://mlsanity.com/paper/2502.06301/) (2025)
- [Deep Reinforcement Learning-Assisted Automated Operator Portfolio for Constrained Multi-objective Optimization](https://mlsanity.com/paper/2603.16401/) (2026)
- [Safe and Robust Neural Policy Learning with Statistical Verification for Sim-to-Real Deployment in Robotics](https://mlsanity.com/paper/2608.06481/) (2026)
- [Covy: An AI-powered Robot with a Compound Vision System for Detecting Breaches in Social Distancing](https://mlsanity.com/paper/2207.06847/) (2022)
- [A Review of Deep Reinforcement Learning in Serverless Computing: Function Scheduling and Resource Auto-Scaling](https://mlsanity.com/paper/2311.12839/) (2023)
- [Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning](https://mlsanity.com/paper/2405.01615/) (2024)
- [Visuospatial navigation from the bottom-up: without vestibular integration, distance prediction, or maps](https://mlsanity.com/paper/2407.13535/) (2024)
- [EnhancedRL: An Enhanced-State Reinforcement Learning Algorithm for Multi-Task Fusion in Recommender Systems](https://mlsanity.com/paper/2409.11678/) (2024)
- [Benchmarking MOEAs for solving continuous multi-objective RL problems](https://mlsanity.com/paper/2505.13726/) (2025)
- [Toward Safety-First Human-Like Decision Making for Autonomous Vehicles in Time-Varying Traffic Flow](https://mlsanity.com/paper/2506.14502/) (2025)
- [Optimal Design of Experiment for Electrochemical Parameter Identification of Li-ion Battery via Deep Reinforcement Learning](https://mlsanity.com/paper/2506.19146/) (2025)
- [Harnessing Bounded-Support Evolution Strategies for Policy Refinement](https://mlsanity.com/paper/2511.09923/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) and Evolution Strategies (ESs) have surpassed human-level control in many sequential decision-making problems, yet many open challenges still exist. To get insights into the strengths and weaknesses of DRL versus ESs, an analysis of their respective capabilities and limitations is provided. After presenting their fundamental concepts and algorithms, a comparison is provided on key aspects such as scalability, exploration, adaptation to dynamic environments, and multi-agent learning. Then, the benefits of hybrid algorithms that combine concepts from DRL and ESs are highlighted. Finally, to have an indication about how they compare in real-world applications, a survey of the literature for the set of applications they support is provided.