# A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning

arXiv: https://arxiv.org/abs/2201.00129 · HTML: https://mlsanity.com/paper/2201.00129/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2022.

## Built on by (most cited first, 13 of 13)

- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation](https://mlsanity.com/paper/2210.17375/) (2022)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Making Reinforcement Learning Work on Swimmer](https://mlsanity.com/paper/2208.07587/) (2022)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning](https://mlsanity.com/paper/2501.13883/) (2025)
- [Evolutionary Reinforcement Learning via Cooperative Coevolution](https://mlsanity.com/paper/2404.14763/) (2024)
- [Surrogate-Assisted Evolutionary Reinforcement Learning Based on Autoencoder and Hyperbolic Neural Network](https://mlsanity.com/paper/2505.19423/) (2025)
- [Evolutionary Enhanced Multi-Agent Reinforcement Learning for Cooperative Air Combat](https://mlsanity.com/paper/2605.25091/) (2026)
- [Genetic Drift Regularization: on preventing Actor Injection from breaking Evolution Strategies](https://mlsanity.com/paper/2405.04322/) (2024)

## Abstract (arXiv)

The integration of Reinforcement Learning (RL) and Evolutionary Algorithms (EAs) aims at simultaneously exploiting the sample efficiency as well as the diversity and robustness of the two paradigms. Recently, hybrid learning frameworks based on this principle have achieved great success in various challenging robot control tasks. However, in these methods, policies from the genetic population are evaluated via interactions with the real environments, limiting their applicability in computationally expensive problems. In this work, we propose Surrogate-assisted Controller (SC), a novel and efficient module that can be integrated into existing frameworks to alleviate the computational burden of EAs by partially replacing the expensive policy evaluation. The key challenge in applying this module is to prevent the optimization process from being misled by the possible false minima introduced by the surrogate. To address this issue, we present two strategies for SC to control the workflow of hybrid frameworks. Experiments on six continuous control tasks from the OpenAI Gym platform show that SC can not only significantly reduce the cost of fitness evaluations, but also boost the performance of the original hybrid frameworks with collaborative learning and evolutionary processes.