# Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles

arXiv: https://arxiv.org/abs/2010.14641 · HTML: https://mlsanity.com/paper/2010.14641/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [State Entropy Maximization with Random Encoders for Efficient Exploration](https://mlsanity.com/paper/2102.09430/) (2021)
- [Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies](https://mlsanity.com/paper/2111.02552/) (2021)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration](https://mlsanity.com/paper/2305.19476/) (2023)
- [Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space](https://mlsanity.com/paper/2102.09812/) (2021)
- [COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL](https://mlsanity.com/paper/2310.07220/) (2023)
- [On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economics](https://mlsanity.com/paper/2310.19527/) (2023)
- [Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models](https://mlsanity.com/paper/2604.25416/) (2026)
- [Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks](https://mlsanity.com/paper/2205.09117/) (2022)
- [Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling](https://mlsanity.com/paper/2410.04988/) (2024)
- [Learning Soft Robotic Dynamics with Active Exploration](https://mlsanity.com/paper/2510.27428/) (2025)
- [A Case for Validation Buffer in Pessimistic Actor-Critic](https://mlsanity.com/paper/2403.01014/) (2024)

## Abstract (arXiv)

Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective. This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives. In sparse and hard to explore environments we achieve an average improvement of over 30%.