# Behavior-based Neuroevolutionary Training in Reinforcement Learning

arXiv: https://arxiv.org/abs/2105.07960 · HTML: https://mlsanity.com/paper/2105.07960/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Evolution-Guided Policy Gradient in Reinforcement Learning](https://mlsanity.com/paper/1805.07917/) (2018)
- [Off-Policy Actor-Critic with Shared Experience Replay](https://mlsanity.com/paper/1909.11583/) (2019)
- [Collaborative Evolutionary Reinforcement Learning](https://mlsanity.com/paper/1905.00976/) (2019)
- [Population-Guided Parallel Policy Search for Reinforcement Learning](https://mlsanity.com/paper/2001.02907/) (2020)
- [Differentiable Genetic Programming](https://mlsanity.com/paper/1611.04766/) (2016)
- [Expected Improvement versus Predicted Value in Surrogate-Based Optimization](https://mlsanity.com/paper/2001.02957/) (2020)
- [The True Destination of EGO is Multi-local Optimization](https://mlsanity.com/paper/1704.05724/) (2017)
- [Data-efficient Neuroevolution with Kernel-Based Surrogate Models](https://mlsanity.com/paper/1804.05364/) (2018)
- [Improving NeuroEvolution Efficiency by Surrogate Model-based Optimization with Phenotypic Distance Kernels](https://mlsanity.com/paper/1902.03419/) (2019)
- [Surrogate Models for Enhancing the Efficiency of Neuroevolution in Reinforcement Learning](https://mlsanity.com/paper/1907.09300/) (2019)
- [Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1910.12824/) (2019)

## Built on by (most cited first, 3 of 3)

- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)

## Abstract (arXiv)

In addition to their undisputed success in solving classical optimization problems, neuroevolutionary and population-based algorithms have become an alternative to standard reinforcement learning methods. However, evolutionary methods often lack the sample efficiency of standard value-based methods that leverage gathered state and value experience. If reinforcement learning for real-world problems with significant resource cost is considered, sample efficiency is essential. The enhancement of evolutionary algorithms with experience exploiting methods is thus desired and promises valuable insights. This work presents a hybrid algorithm that combines topology-changing neuroevolutionary optimization with value-based reinforcement learning. We illustrate how the behavior of policies can be used to create distance and loss functions, which benefit from stored experiences and calculated state values. They allow us to model behavior and perform a directed search in the behavior space by gradient-free evolutionary algorithms and surrogate-based optimization. For this purpose, we consolidate different methods to generate and optimize agent policies, creating a diverse population. We exemplify the performance of our algorithm on standard benchmarks and a purpose-built real-world problem. Our results indicate that combining methods can enhance the sample efficiency and learning speed for evolutionary approaches.