mlsanityData to 14 Oct 2026

Smart Exploration in Reinforcement Learning using Bounded Uncertainty Models

arXiv 2504.05978 ↗First posted 8 Apr 2025cs.LG cs.SY eess.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning (RL) is a powerful framework for decision-making in uncertain environments, but it often requires large amounts of data to learn an optimal policy. We address this challenge by incorporating prior model knowledge to guide exploration and accelerate the learning process. Specifically, we assume access to a model set that contains the true transition kernel and reward function. We optimize over this model set to obtain upper and lower bounds on the Q-function, which are then used to guide the exploration of the agent. We provide theoretical guarantees on the convergence of the Q-function to the optimal Q-function under the proposed class of exploring policies. Furthermore, we also introduce a data-driven regularized version of the model set optimization problem that ensures the convergence of the class of exploring policies to the optimal policy. Lastly, we show that when the model set has a specific structure, namely the bounded-parameter MDP (BMDP) framework, the regularized model set optimization problem becomes convex and simple to implement. In this setting, we also prove finite-time convergence to the optimal policy under mild assumptions. We demonstrate the effectiveness of the proposed exploration strategy, which we call BUMEX (Bounded Uncertainty Model-based Exploration), in a simulation study. The results indicate that the proposed method can significantly accelerate learning in benchmark examples. A toolbox is available at https://github.com/JvHulst/BUMEX.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 23 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

8 of 23 references.

References · 8

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016OpenAI Gymcited by 1,736
  3. 2018Is Q-learning Provably Efficient?cited by 487
  4. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  5. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
  6. 2022From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusescited by 22
  7. 2022Robust Anytime Learning of Markov Decision Processescited by 16
  8. 2023A Contracting Dynamical System Perspective toward Interval Markov Decision Processescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Model-Free Active Exploration in Reinforcement Learning
  2. 2024Bounded Exploration with World Model Uncertainty in Soft Actor-Critic Reinforcement Learning Algorithm
  3. 2025On Efficient Bayesian Exploration in Model-Based Reinforcement Learning
  4. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
  5. 2025Sample-efficient and Scalable Exploration in Continuous-Time RL
  6. 2025Look Before Leap: Look-Ahead Planning with Uncertainty in Reinforcement Learning
  7. 2026All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning
  8. 2024Prior-dependent analysis of posterior sampling reinforcement learning with function approximation