Exploration in Model-based Reinforcement Learning with Randomized Reward
Not on the map. Why?
Abstract (arXiv)
Model-based Reinforcement Learning (MBRL) has been widely adapted due to its sample efficiency. However, existing worst-case regret analysis typically requires optimistic planning, which is not realistic in general. In contrast, motivated by the theory, empirical study utilizes ensemble of models, which achieve state-of-the-art performance on various testing environments. Such deviation between theory and empirical study leads us to question whether randomized model ensemble guarantee optimism, and hence the optimal worst-case regret? This paper partially answers such question from the perspective of reward randomization, a scarcely explored direction of exploration with MBRL. We show that under the kernelized linear regulator (KNR) model, reward randomization guarantees a partial optimism, which further yields a near-optimal worst-case regret in terms of the number of interactions. We further extend our theory to generalized function approximation and identified conditions for reward randomization to attain provably efficient exploration. Correspondingly, we propose concrete examples of efficient reward randomization. To the best of our knowledge, our analysis establishes the first worst-case regret analysis on randomized MBRL with function approximation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 9 Jan 2023.
Builds on
28 of 37 references.
References · 28
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2018World Modelscited by 1,444
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
- 2019Model-Based Reinforcement Learning for Ataricited by 495
- 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
- 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
- 2019Self-Supervised Exploration via Disagreementcited by 263
- 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
- 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
- 2019Provably Efficient Exploration in Policy Optimizationcited by 203
16 more
- 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
- 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
- 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
- 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
- 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
- 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
- 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
- 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
- 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
- 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
- 2020Active Learning for Nonlinear System Identification with Guaranteescited by 78
- 2019Worst-Case Regret Bounds for Exploration via Randomized Value Functionscited by 57
- 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
- 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
- 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
- 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
Built on by
Not cited yet.