mlsanityData to 14 Oct 2026

Goal-Space Planning with Subgoal Models

arXiv 2206.02902 ↗First posted 6 Jun 2022cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Background planning with learned models is often worse than model-free alternatives, such as Double DQN, even though the former uses significantly more memory and computation. The fundamental problem is that learned models can be inaccurate and often generate invalid states, especially when iterated many steps. In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models. This goal-space planning (GSP) approach is more computationally efficient, naturally incorporates temporal abstraction for faster long-horizon planning and avoids learning the transition dynamics entirely. We show that our GSP algorithm can propagate value from an abstract space in a manner that helps a variety of base learners learn significantly faster in different domains.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

31 of 56 references.

References · 31

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  5. 2016Value Iteration Networkscited by 160
  6. 2019Planning with Goal-Conditioned Policiescited by 149
  7. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 140
  8. 2019When to use parametric models in reinforcement learning?cited by 118
  9. 2017Value Prediction Networkcited by 117
  10. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  11. 2016The Predictron: End-To-End Learning and Planningcited by 93
  12. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 69
19 more
  1. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
  2. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  3. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  4. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  5. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  6. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  7. 2021Learning Neuro-Symbolic Relational Transition Models for Bilevel Planningcited by 43
  8. 2016Unifying task specification in reinforcement learningcited by 41
  9. 2020What can I do here? A Theory of Affordances in Reinforcement Learningcited by 35
  10. 2020Sparse Graphical Memory for Robust Planningcited by 31
  11. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 28
  12. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  13. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  14. 2020Forethought and Hindsight in Credit Assignmentcited by 16
  15. 2022Reward-Respecting Subtasks for Model-Based Reinforcement Learningcited by 16
  16. 2019Planning with Expectation Modelscited by 12
  17. 2021Average-Reward Learning and Planning with Optionscited by 8
  18. 2021DisTop: Discovering a Topological representation to learn diverse and rewarding skillscited by 7
  19. 2019Hill Climbing on Value Estimates for Search-control in Dynacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 2; 2026: 6.

12 papers.

  1. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  2. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  3. 2026Compositional Planning with Jumpy World Modelscited by 7
  4. 2023Learning with a Mole: Transferable latent spatial representations for navigation without reconstructioncited by 6
  5. 2022Auxiliary task discovery through generate-and-testcited by 2
  6. 2024Rejecting Hallucinated State Targets during Planningcited by 1
  7. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  8. 2026Position: Deployed Reinforcement Learning should be Continualcited by 1
  9. 2024A New View on Planning in Online Reinforcement Learningcited by 0
  10. 2026Addressing Terminal Constraints in Data-Driven Demand Response Schedulingcited by 0
  11. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0
  12. 2026Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Criticcited by 0