# DORA The Explorer: Directed Outreaching Reinforcement Action-Selection

arXiv: https://arxiv.org/abs/1804.04012 · HTML: https://mlsanity.com/paper/1804.04012/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2018.

## Built on by (most cited first, 27 of 27)

- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses](https://mlsanity.com/paper/2205.07704/) (2022)
- [Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning](https://mlsanity.com/paper/2408.03029/) (2024)
- [Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model](https://mlsanity.com/paper/2103.08255/) (2021)
- [Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning](https://mlsanity.com/paper/2105.06696/) (2021)
- [Foundation Models for Semantic Novelty in Reinforcement Learning](https://mlsanity.com/paper/2211.04878/) (2022)
- [Exploration-Driven Generative Interactive Environments](https://mlsanity.com/paper/2504.02515/) (2025)
- [Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping](https://mlsanity.com/paper/2209.07288/) (2022)
- [On the Importance of Reward Design in Reinforcement Learning-based Dynamic Algorithm Configuration: A Case Study on OneMax with (1+(λ,λ))-GA](https://mlsanity.com/paper/2502.20265/) (2025)
- [Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying](https://mlsanity.com/paper/2606.00151/) (2026)
- [Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments](https://mlsanity.com/paper/2211.10515/) (2022)
- [Neuro-Inspired Fragmentation and Recall to Overcome Catastrophic Forgetting in Curiosity](https://mlsanity.com/paper/2310.17537/) (2023)
- [Back-stepping Experience Replay with Application to Model-free Reinforcement Learning for a Soft Snake Robot](https://mlsanity.com/paper/2401.11372/) (2024)
- [Learning user-defined sub-goals using memory editing in reinforcement learning](https://mlsanity.com/paper/2205.00399/) (2022)
- [Cyclophobic Reinforcement Learning](https://mlsanity.com/paper/2308.15911/) (2023)
- [Exploration by Random Reward Perturbation](https://mlsanity.com/paper/2506.08737/) (2025)
- [Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GA](https://mlsanity.com/paper/2512.03805/) (2025)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Learning from Demonstration without Demonstrations](https://mlsanity.com/paper/2106.09203/) (2021)
- [Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction](https://mlsanity.com/paper/2110.12081/) (2021)
- [AutoDIME: Automatic Design of Interesting Multi-Agent Environments](https://mlsanity.com/paper/2203.02481/) (2022)
- [GAN-based Intrinsic Exploration For Sample Efficient Reinforcement Learning](https://mlsanity.com/paper/2206.14256/) (2022)
- [Scope Loss for Imbalanced Classification and RL Exploration](https://mlsanity.com/paper/2308.04024/) (2023)
- [Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2404.12999/) (2024)
- [Optimizing TD3 for 7-DOF Robotic Arm Grasping: Overcoming Suboptimality with Exploration-Enhanced Contrastive Learning](https://mlsanity.com/paper/2408.14009/) (2024)
- [Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning](https://mlsanity.com/paper/2412.03767/) (2024)
- [Learning What Matters: Adaptive Information-Theoretic Objectives for Robot Exploration](https://mlsanity.com/paper/2605.12084/) (2026)

## Abstract (arXiv)

Exploration is a fundamental aspect of Reinforcement Learning, typically implemented using stochastic action-selection. Exploration, however, can be more efficient if directed toward gaining new world knowledge. Visit-counters have been proven useful both in practice and in theory for directed exploration. However, a major limitation of counters is their locality. While there are a few model-based solutions to this shortcoming, a model-free approach is still missing. We propose E-values, a generalization of counters that can be used to evaluate the propagating exploratory value over state-action trajectories. We compare our approach to commonly used RL techniques, and show that using E-values improves learning and performance over traditional counters. We also show how our method can be implemented with function approximation to efficiently learn continuous MDPs. We demonstrate this by showing that our approach surpasses state of the art performance in the Freeway Atari 2600 game.