mlsanityData to 7 Oct 2026

A New View on Planning in Online Reinforcement Learning

arXiv 2406.01562 ↗First posted 3 Jun 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Background planning with learned models is often worse than model-free alternatives, such as Double DQN, even though the former uses significantly more memory and computation. The fundamental problem is that learned models can be inaccurate and often generate invalid states, especially when iterated many steps. In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models. This goal-space planning (GSP) approach is more computationally efficient, naturally incorporates temporal abstraction for faster long-horizon planning and avoids learning the transition dynamics entirely. We show that our GSP algorithm can propagate value from an abstract space in a manner that helps a variety of base learners learn significantly faster in different domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 61 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

28 of 61 references.

References · 28

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  6. 2016Value Iteration Networkscited by 158
  7. 2019Planning with Goal-Conditioned Policiescited by 144
  8. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  9. 2019When to use parametric models in reinforcement learning?cited by 118
  10. 2017Value Prediction Networkcited by 117
  11. 2016The Predictron: End-To-End Learning and Planningcited by 92
  12. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
16 more
  1. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  2. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  3. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  4. 2021Learning Neuro-Symbolic Relational Transition Models for Bilevel Planningcited by 43
  5. 2020Sparse Graphical Memory for Robust Planningcited by 30
  6. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
  7. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  8. 2013Hierarchical Solution of Markov Decision Processes using Macro-actionscited by 19
  9. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  10. 2018Finding Options that Minimize Planning Timecited by 17
  11. 2020Forethought and Hindsight in Credit Assignmentcited by 16
  12. 2019Planning with Expectation Modelscited by 12
  13. 2022Goal-Space Planning with Subgoal Modelscited by 12
  14. 2021Average-Reward Learning and Planning with Optionscited by 8
  15. 2019Planning in Hierarchical Reinforcement Learning: Guarantees for Using Local Policiescited by 5
  16. 2015Constructing Abstraction Hierarchies Using a Skill-Symbol Loopcited by 3

Built on by

Not cited yet.