mlsanityData to 7 Oct 2026

Planning with Expectation Models for Control

arXiv 2104.08543 ↗First posted 17 Apr 2021cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In model-based reinforcement learning (MBRL), Wan et al. (2019) showed conditions under which the environment model could produce the expectation of the next feature vector rather than the full distribution, or a sample thereof, with no loss in planning performance. Such expectation models are of interest when the environment is stochastic and non-stationary, and the model is approximate, such as when it is learned using function approximation. In these cases a full distribution model may be impractical and a sample model may be either more expensive computationally or of high variance. Wan et al. considered only planning for prediction to evaluate a fixed policy. In this paper, we treat the control case - planning to improve and find a good approximate policy. We prove that planning with an expectation model must update a state-value function, not an action-value function as previously suggested (e.g., Sorg & Singh, 2010). This opens the question of how planning influences action selections. We consider three strategies for this and present general MBRL algorithms for each. We identify the strengths and weaknesses of these algorithms in computational experiments. Our algorithms and experiments are the first to treat MBRL with expectation models in a general setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 78 references that are in the map. First posted on arXiv 17 Apr 2021.

Builds on

26 of 78 references.

References · 26

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018World Modelscited by 1,444
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2019Model-Based Reinforcement Learning for Ataricited by 495
  11. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  12. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
14 more
  1. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  2. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  3. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  4. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  5. 2019When to use parametric models in reinforcement learning?cited by 118
  6. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 40
  7. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 25
  8. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  9. 2020Efficient Planning in Large MDPs with Weak Linear Function Approximationcited by 17
  10. 2018The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spacescited by 15
  11. 2019Planning with Expectation Modelscited by 12
  12. 2020The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learningcited by 11
  13. 2016A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Gamescited by 9
  14. 2019Hill Climbing on Value Estimates for Search-control in Dynacited by 6

Built on by

Not cited yet.