# Directed Exploration for Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.07805 · HTML: https://mlsanity.com/paper/1906.07805/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 9 of 9)

- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Efficient Self-Supervised Data Collection for Offline Robot Learning](https://mlsanity.com/paper/2105.04607/) (2021)
- [Directed Exploration in Reinforcement Learning from Linear Temporal Logic](https://mlsanity.com/paper/2408.09495/) (2024)
- [Structure-Enhanced DRL for Optimal Transmission Scheduling](https://mlsanity.com/paper/2212.12704/) (2022)
- [Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines](https://mlsanity.com/paper/2507.05561/) (2025)
- [Structure-Enhanced Deep Reinforcement Learning for Optimal Transmission Scheduling](https://mlsanity.com/paper/2211.10827/) (2022)
- [Delayed Geometric Discounts: An Alternative Criterion for Reinforcement Learning](https://mlsanity.com/paper/2209.12483/) (2022)
- [Directed evolution algorithm drives neural prediction](https://mlsanity.com/paper/2512.01362/) (2025)

## Abstract (arXiv)

Efficient exploration is necessary to achieve good sample efficiency for reinforcement learning in general. From small, tabular settings such as gridworlds to large, continuous and sparse reward settings such as robotic object manipulation tasks, exploration through adding an uncertainty bonus to the reward function has been shown to be effective when the uncertainty is able to accurately drive exploration towards promising states. However reward bonuses can still be inefficient since they are non-stationary, which means that we must wait for function approximators to catch up and converge again when uncertainties change. We propose the idea of directed exploration, that is learning a goal-conditioned policy where goals are simply other states, and using that to directly try to reach states with large uncertainty. The goal-conditioned policy is independent of uncertainty and is thus stationary. We show in our experiments how directed exploration is more efficient at exploration and more robust to how the uncertainty is computed than adding bonuses to rewards.