# Temporally-Extended ε-Greedy Exploration

arXiv: https://arxiv.org/abs/2006.01782 · HTML: https://mlsanity.com/paper/2006.01782/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2020.

## Built on by (most cited first, 29 of 29)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximation](https://mlsanity.com/paper/2206.09421/) (2022)
- [Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning](https://mlsanity.com/paper/2102.13515/) (2021)
- [When should agents explore?](https://mlsanity.com/paper/2108.11811/) (2021)
- [TAAC: Temporally Abstract Actor-Critic for Continuous Control](https://mlsanity.com/paper/2104.06521/) (2021)
- [When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL](https://mlsanity.com/paper/2406.01163/) (2024)
- [Machine Learning for Mechanical Ventilation Control](https://mlsanity.com/paper/2102.06779/) (2021)
- [LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Framework](https://mlsanity.com/paper/2310.03342/) (2023)
- [Successor-Predecessor Intrinsic Exploration](https://mlsanity.com/paper/2305.15277/) (2023)
- [Random Latent Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/2407.13755/) (2024)
- [FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control](https://mlsanity.com/paper/2604.04539/) (2026)
- [SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2205.13528/) (2022)
- [Bayesian Q-learning With Imperfect Expert Demonstrations](https://mlsanity.com/paper/2210.01800/) (2022)
- [Policy Gradient With Serial Markov Chain Reasoning](https://mlsanity.com/paper/2210.06766/) (2022)
- [Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks](https://mlsanity.com/paper/2403.01636/) (2024)
- [Delphos: A reinforcement learning framework for assisting discrete choice model specification](https://mlsanity.com/paper/2506.06410/) (2025)
- [Temporally Extended Successor Representations](https://mlsanity.com/paper/2209.12331/) (2022)
- [RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design](https://mlsanity.com/paper/2512.05403/) (2025)
- [Learning Routines for Effective Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2106.02943/) (2021)
- [Machine Learning for Mechanical Ventilation Control (Extended Abstract)](https://mlsanity.com/paper/2111.10434/) (2021)
- [Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.10803/) (2022)
- [Combining Evolutionary Search with Behaviour Cloning for Procedurally Generated Content](https://mlsanity.com/paper/2207.14772/) (2022)
- [RadDQN: a Deep Q Learning-based Architecture for Finding Time-efficient Minimum Radiation Exposure Pathway](https://mlsanity.com/paper/2402.00468/) (2024)
- [Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning](https://mlsanity.com/paper/2412.03767/) (2024)
- [Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learning](https://mlsanity.com/paper/2509.20766/) (2025)
- [PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model](https://mlsanity.com/paper/2608.06794/) (2026)

## Abstract (arXiv)

Recent work on exploration in reinforcement learning (RL) has led to a series of increasingly complex solutions to the problem. This increase in complexity often comes at the expense of generality. Recent empirical studies suggest that, when applied to a broader set of domains, some sophisticated exploration methods are outperformed by simpler counterparts, such as ε-greedy. In this paper we propose an exploration algorithm that retains the simplicity of ε-greedy while reducing dithering. We build on a simple hypothesis: the main limitation of ε-greedy exploration is its lack of temporal persistence, which limits its ability to escape local optima. We propose a temporally extended form of ε-greedy that simply repeats the sampled action for a random duration. It turns out that, for many duration distributions, this suffices to improve exploration on a large set of domains. Interestingly, a class of distributions inspired by ecological models of animal foraging behaviour yields particularly strong performance.