mlsanityData to 14 Oct 2026

Evolutionary Policy Optimization

arXiv 2503.19037 ↗First posted 24 Mar 2025cs.LG cs.AI cs.ROcited by 4

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

On-policy reinforcement learning (RL) algorithms are widely used for their strong asymptotic performance and training stability, but they struggle to scale with larger batch sizes, as additional parallel environments yield redundant data due to limited policy-induced diversity. In contrast, Evolutionary Algorithms (EAs) scale naturally and encourage exploration via randomized population-based search, but are often sample-inefficient. We propose Evolutionary Policy Optimization (EPO), a hybrid algorithm that combines the scalability and diversity of EAs with the performance and stability of policy gradients. EPO maintains a population of agents conditioned on latent variables, shares actor-critic network parameters for coherence and memory efficiency, and aggregates diverse experiences into a master agent. Across tasks in dexterous manipulation, legged locomotion, and classic control, EPO outperforms state-of-the-art baselines in sample efficiency, asymptotic performance, and scalability.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 51 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

34 of 51 references.

References · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  12. 2018DeepMind Control Suitecited by 956
22 more
  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  3. 2019Solving Rubik's Cube with a Robot Handcited by 734
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  5. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  6. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  7. 2017Population Based Training of Neural Networkscited by 367
  8. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  9. 2023Extreme Parkour with Legged Robotscited by 164
  10. 2023Robot Parkour Learningcited by 149
  11. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  12. 2018Evolution-Guided Policy Gradient in Reinforcement Learningcited by 83
  13. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  14. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  15. 2019Weight Agnostic Neural Networkscited by 71
  16. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 58
  17. 2018Structured Evolution with Compact Architectures for Scalable Policy Optimizationcited by 48
  18. 2019Collaborative Evolutionary Reinforcement Learningcited by 37
  19. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 29
  20. 2018Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Ataricited by 28
  21. 2024SAPG: Split and Aggregate Policy Gradientscited by 21
  22. 2023Rethinking Population-assisted Off-policy Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learningcited by 1
  2. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 1
  3. 2025Harnessing Bounded-Support Evolution Strategies for Policy Refinementcited by 0
  4. 2026Jevolution: A Framework for Studying Communication and Collective Dynamics in AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolutionary Policy Optimization
  2. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms
  3. 2024Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning
  4. 2024Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts
  5. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
  6. 2025Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots
  7. 2024SPO: Sequential Monte Carlo Policy Optimisation
  8. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization