# Explicit Explore-Exploit Algorithms in Continuous State Spaces

arXiv: https://arxiv.org/abs/1911.00617 · HTML: https://mlsanity.com/paper/1911.00617/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2019.

## Built on by (most cited first, 19 of 19)

- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [Conditional Mutual Information for Disentangled Representations in Reinforcement Learning](https://mlsanity.com/paper/2305.14133/) (2023)
- [Grounding Video Models to Actions through Goal Conditioned Exploration](https://mlsanity.com/paper/2411.07223/) (2024)
- [Multi-Stage Episodic Control for Strategic Exploration in Text Games](https://mlsanity.com/paper/2201.01251/) (2022)
- [A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs](https://mlsanity.com/paper/2306.03236/) (2023)
- [Adaptive Discretization in Online Reinforcement Learning](https://mlsanity.com/paper/2110.15843/) (2021)
- [Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning](https://mlsanity.com/paper/2201.09765/) (2022)
- [Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks](https://mlsanity.com/paper/2110.02414/) (2021)
- [Epistemic Monte Carlo Tree Search](https://mlsanity.com/paper/2210.13455/) (2022)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Explicit Explore, Exploit, or Escape (E4): near-optimal safety-constrained reinforcement learning in polynomial time](https://mlsanity.com/paper/2111.07395/) (2021)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments](https://mlsanity.com/paper/2211.10515/) (2022)
- [Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2310.05723/) (2023)
- [A Survey of Historical Learning: Learning Models with Learning History](https://mlsanity.com/paper/2303.12992/) (2023)
- [Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2411.01396/) (2024)
- [Learning to Price: Interpretable Attribute-Level Models for Dynamic Markets](https://mlsanity.com/paper/2602.00188/) (2026)

## Abstract (arXiv)

We present a new model-based algorithm for reinforcement learning (RL) which consists of explicit exploration and exploitation phases, and is applicable in large or infinite state spaces. The algorithm maintains a set of dynamics models consistent with current experience and explores by finding policies which induce high disagreement between their state predictions. It then exploits using the refined set of models or experience gathered during exploration. We show that under realizability and optimal planning assumptions, our algorithm provably finds a near-optimal policy with a number of samples that is polynomial in a structural complexity measure which we show to be low in several natural settings. We then give a practical approximation using neural networks and demonstrate its performance and sample efficiency in practice.