mlsanityData to 14 Oct 2026

Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

arXiv 1712.06560 ↗First posted 18 Dec 2017cs.AIcited by 107

Not on the map. Why?

Abstract (arXiv)

Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 27; 2023: 19; 2024: 10; 2025: 20; 2026: 8.

107 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  2. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  3. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  4. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 73
  5. 2021Differentiable Quality Diversitycited by 72
  6. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
  7. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 56
  8. 2021Variational Quantum Reinforcement Learning via Evolutionary Optimizationcited by 50
  9. 2022Reinforced Genetic Algorithm for Structure-based Drug Designcited by 46
  10. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 41
  11. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  12. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 38
38 more
  1. 2021Illuminating Diverse Neural Cellular Automata for Level Generationcited by 35
  2. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  3. 2023pyribs: A Bare-Bones Python Library for Quality Diversity Optimizationcited by 30
  4. 2023Evolutionary Reinforcement Learning: A Surveycited by 29
  5. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 28
  6. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  7. 2022Covariance Matrix Adaptation MAP-Annealingcited by 25
  8. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 23
  9. 2022Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discoverycited by 23
  10. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 23
  11. 2025HeurAgenix: Leveraging LLMs for Solving Complex Combinatorial Optimization Challengescited by 21
  12. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  13. 2022Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectoriescited by 20
  14. 2023Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learningcited by 20
  15. 2025Pareto Set Learning for Multi-Objective Reinforcement Learningcited by 20
  16. 2021Discovering Diverse Athletic Jumping Strategiescited by 16
  17. 2021Derivative-Free Reinforcement Learning: A Reviewcited by 15
  18. 2021MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Corescited by 15
  19. 2022TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learningcited by 15
  20. 2022Accelerated Quality-Diversity through Massive Parallelismcited by 14
  21. 2023Games for Artificial Intelligence Research: A Review and Perspectivescited by 14
  22. 2021Sparse Reward Exploration via Novelty Search and Emitterscited by 13
  23. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  24. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  25. 2022Heterogeneous Multi-agent Zero-Shot Coordination by Coevolutioncited by 12
  26. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  27. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  28. 2022Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealingcited by 10
  29. 2023Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimizationcited by 10
  30. 2021Model-free Policy Learning with Reward Gradientscited by 9
  31. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  32. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  33. 2021Direct-Search for a Class of Stochastic Min-Max Problemscited by 7
  34. 2022On-the-fly Strategy Adaptation for ad-hoc Agent Coordinationcited by 7
  35. 2022Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidancecited by 7
  36. 2024Large Language Models as In-context AI Generators for Quality-Diversitycited by 7
  37. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
  38. 2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7

50 most cited of 107 All 107 (JSON)