# Model-Based Active Exploration

arXiv: https://arxiv.org/abs/1810.12162 · HTML: https://mlsanity.com/paper/1810.12162/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 30 of 105)

- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Discovering and Achieving Goals via World Models](https://mlsanity.com/paper/2110.09514/) (2021)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Semantic Exploration from Language Abstractions and Pretrained Representations](https://mlsanity.com/paper/2204.05080/) (2022)
- [Optimizing Sequential Experimental Design with Deep Reinforcement Learning](https://mlsanity.com/paper/2202.00821/) (2022)
- [MADE: Exploration via Maximizing Deviation from Explored Regions](https://mlsanity.com/paper/2106.10268/) (2021)
- [Learning more skills through optimistic exploration](https://mlsanity.com/paper/2107.14226/) (2021)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [Optimistic Active Exploration of Dynamical Systems](https://mlsanity.com/paper/2306.12371/) (2023)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [ASID: Active Exploration for System Identification in Robotic Manipulation](https://mlsanity.com/paper/2404.12308/) (2024)
- [How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation](https://mlsanity.com/paper/2102.04399/) (2021)
- [Grounding Video Models to Actions through Goal Conditioned Exploration](https://mlsanity.com/paper/2411.07223/) (2024)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion](https://mlsanity.com/paper/2310.17634/) (2023)
- [Embodied AI: From LLMs to World Models](https://mlsanity.com/paper/2509.20021/) (2025)
- [Curiosity-Driven Exploration via Latent Bayesian Surprise](https://mlsanity.com/paper/2104.07495/) (2021)
- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [A Differential Entropy Estimator for Training Neural Networks](https://mlsanity.com/paper/2202.06618/) (2022)
- [A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs](https://mlsanity.com/paper/2306.03236/) (2023)
- [Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation](https://mlsanity.com/paper/2107.01825/) (2021)
- [A Bayesian Framework for Digital Twin-Based Control, Monitoring, and Data Collection in Wireless Systems](https://mlsanity.com/paper/2212.01351/) (2022)
- [World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry](https://mlsanity.com/paper/2604.01985/) (2026)

## Abstract (arXiv)

Efficient exploration is an unsolved problem in Reinforcement Learning which is usually addressed by reactively rewarding the agent for fortuitously encountering novel situations. This paper introduces an efficient active exploration algorithm, Model-Based Active eXploration (MAX), which uses an ensemble of forward models to plan to observe novel events. This is carried out by optimizing agent behaviour with respect to a measure of novelty derived from the Bayesian perspective of exploration, which is estimated using the disagreement between the futures predicted by the ensemble members. We show empirically that in semi-random discrete environments where directed exploration is critical to make progress, MAX is at least an order of magnitude more efficient than strong baselines. MAX scales to high-dimensional continuous environments where it builds task-agnostic models that can be used for any downstream task.