mlsanityData to 13 Oct 2026

On the Benefits of Leveraging Structural Information in Planning Over the Learned Model

arXiv 2303.08856 ↗First posted 15 Mar 2023cs.LG cs.SY eess.SYcited by 0

Not on the map. Why?

Abstract (arXiv)

Model-based Reinforcement Learning (RL) integrates learning and planning and has received increasing attention in recent years. However, learning the model can incur a significant cost (in terms of sample complexity), due to the need to obtain a sufficient number of samples for each state-action pair. In this paper, we investigate the benefits of leveraging structural information about the system in terms of reducing sample complexity. Specifically, we consider the setting where the transition probability matrix is a known function of a number of structural parameters, whose values are initially unknown. We then consider the problem of estimating those parameters based on the interactions with the environment. We characterize the difference between the Q estimates and the optimal Q value as a function of the number of samples. Our analysis shows that there can be a significant saving in sample complexity by leveraging structural information about the model. We illustrate the findings by considering several problems including controlling a queuing system with heterogeneous servers, and seeking an optimal path in a stochastic windy gridworld.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 9 of its 28 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

9 of 28 references.

References · 9

  1. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 126
  2. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  3. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  4. 2017Variance Reduced Value Iteration and Faster Algorithms for Solving Markov Decision Processescited by 57
  5. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  6. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  7. 2017Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processescited by 29
  8. 2020Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trialscited by 20
  9. 2021Sample-Efficient Reinforcement Learning for Linearly-Parameterized MDPs with a Generative Modelcited by 18

Built on by

Not cited yet.