mlsanityData to 14 Oct 2026

Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling

arXiv 2410.04988 ↗First posted 7 Oct 2024cs.LG cs.ROcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Learning complex robot behavior through interactions with the environment necessitates principled exploration. Effective strategies should prioritize exploring regions of the state-action space that maximize rewards, with optimistic exploration emerging as a promising direction aligned with this idea and enabling sample-efficient reinforcement learning. However, existing methods overlook a crucial aspect: the need for optimism to be informed by a belief connecting the reward and state. To address this, we propose a practical, theoretically grounded approach to optimistic exploration based on Thompson sampling. Our model structure is the first that allows for reasoning about joint uncertainty over transitions and rewards. We apply our method on a set of MuJoCo and VMAS continuous control tasks. Our experiments demonstrate that optimistic exploration significantly accelerates learning in environments with sparse rewards, action penalties, and difficult-to-explore regions. Furthermore, we provide insights into when optimism is beneficial and emphasize the critical role of model uncertainty in guiding exploration.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 50 of its 71 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

50 of 71 references.

References · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  9. 2023Mastering Diverse Domains through World Modelscited by 1,253
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
38 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  2. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2019Model-Based Reinforcement Learning for Ataricited by 502
  6. 2017A Tutorial on Thompson Samplingcited by 433
  7. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  8. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  9. 2017On Kernelized Multi-armed Banditscited by 352
  10. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  11. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  12. 2014Scalable Variational Gaussian Process Classificationcited by 228
  13. 2017Concrete Dropoutcited by 227
  14. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  15. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  16. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  17. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  18. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  19. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  20. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  21. 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 98
  22. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  23. 2022VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learningcited by 68
  24. 2019Data Efficient Reinforcement Learning for Legged Robotscited by 63
  25. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  26. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 60
  27. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 53
  28. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 52
  29. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
  30. 2016On Lower Bounds for Regret in Reinforcement Learningcited by 40
  31. 2019On Online Learning in Kernelized Markov Decision Processescited by 35
  32. 2019Meta-Learning Mean Functions for Gaussian Processescited by 17
  33. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  34. 2017Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processescited by 14
  35. 2020Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensemblescited by 14
  36. 2022Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computationcited by 14
  37. 2023Memory-Based Dual Gaussian Processes for Sequential Learningcited by 11
  38. 2022Exploration via Planning for Information about the Optimal Trajectorycited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  2. 2025No-Regret Thompson Sampling for Finite-Horizon Markov Decision Processes with Gaussian Processescited by 4
  3. 2026Posterior Sampling Reinforcement Learning with Gaussian Processes for Continuous Control: Sublinear Regret Bounds for Unbounded State Spacescited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning
  2. 2024Beyond Optimism: Exploration With Partially Observable Rewards
  3. 2025SOMBRL: Scalable and Optimistic Model-Based RL
  4. 2026SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning
  5. 2026Sampling-Based Safe Reinforcement Learning
  6. 2025General Exploratory Bonus for Optimistic Exploration in RLHF
  7. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learning
  8. 2025On Efficient Bayesian Exploration in Model-Based Reinforcement Learning