# GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

arXiv: https://arxiv.org/abs/1802.05054 · HTML: https://mlsanity.com/paper/1802.05054/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2018.

## Built on by (most cited first, 30 of 40)

- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Multi-Stage Episodic Control for Strategic Exploration in Text Games](https://mlsanity.com/paper/2201.01251/) (2022)
- [Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance](https://mlsanity.com/paper/2206.03787/) (2022)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/2104.05043/) (2021)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [Sparse Reward Exploration via Novelty Search and Emitters](https://mlsanity.com/paper/2102.03140/) (2021)
- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [Asking for Knowledge: Training RL Agents to Query External Knowledge Using Language](https://mlsanity.com/paper/2205.06111/) (2022)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Improving Deep Policy Gradients with Value Function Search](https://mlsanity.com/paper/2302.10145/) (2023)
- [World Models with Hints of Large Language Models for Goal Achieving](https://mlsanity.com/paper/2406.07381/) (2024)
- [Discovering and Exploiting Sparse Rewards in a Learned Behavior Space](https://mlsanity.com/paper/2111.01919/) (2021)
- [Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks](https://mlsanity.com/paper/2210.03325/) (2022)
- [Evolving Populations of Diverse RL Agents with MAP-Elites](https://mlsanity.com/paper/2303.12803/) (2023)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [SPO: Sequential Monte Carlo Policy Optimisation](https://mlsanity.com/paper/2402.07963/) (2024)
- [Evolution Guided Generative Flow Networks](https://mlsanity.com/paper/2402.02186/) (2024)
- [DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time](https://mlsanity.com/paper/2412.08053/) (2024)
- [Robots Learn Increasingly Complex Tasks with Intrinsic Motivation and Automatic Curriculum Learning](https://mlsanity.com/paper/2202.10222/) (2022)
- [Beyond the Boundaries of Proximal Policy Optimization](https://mlsanity.com/paper/2411.00666/) (2024)
- [Multi-Agent Guided Policy Optimization](https://mlsanity.com/paper/2507.18059/) (2025)

## Abstract (arXiv)

In continuous action domains, standard deep reinforcement learning algorithms like DDPG suffer from inefficient exploration when facing sparse or deceptive reward problems. Conversely, evolutionary and developmental methods focusing on exploration like Novelty Search, Quality-Diversity or Goal Exploration Processes explore more robustly but are less efficient at fine-tuning policies using gradient descent. In this paper, we present the GEP-PG approach, taking the best of both worlds by sequentially combining a Goal Exploration Process and two variants of DDPG. We study the learning performance of these components and their combination on a low dimensional deceptive reward problem and on the larger Half-Cheetah benchmark. We show that DDPG fails on the former and that GEP-PG improves over the best DDPG variant in both environments. Supplementary videos and discussion can be found at http://frama.link/gep_pg, the code at http://github.com/flowersteam/geppg.