Subgoal-Guided Policy Heuristic Search with Learned Subgoals
Not on the map. Why?
Abstract (arXiv)
Policy tree search is a family of tree search algorithms that use a policy to guide the search. These algorithms provide guarantees on the number of expansions required to solve a given problem that are based on the quality of the policy. While these algorithms have shown promising results, the process in which they are trained requires complete solution trajectories to train the policy. Search trajectories are obtained during a trial-and-error search process. When the training problem instances are hard, learning can be prohibitively costly, especially when starting from a randomly initialized policy. As a result, search samples are wasted in failed attempts to solve these hard instances. This paper introduces a novel method for learning subgoal-based policies for policy tree search algorithms. The subgoals and policies conditioned on subgoals are learned from the trees that the search expands while attempting to solve problems, including the search trees of failed attempts. We empirically show that our policy formulation and training method improve the sample efficiency of learning a policy and heuristic function in this online setting.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 8 Jun 2025.
Builds on
23 of 38 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Neural Discrete Representation Learningcited by 4,650
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
- 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
- 2018Relational Deep Reinforcement Learningcited by 72
- 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
- 2019An investigation of model-free planningcited by 64
- 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
- 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
- 2018Single-Agent Policy Tree Search With Guaranteescited by 22
11 more
- 2021Policy-Guided Heuristic Search with Guaranteescited by 20
- 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
- 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
- 2022Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Searchcited by 12
- 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
- 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
- 2021DisTop: Discovering a Topological representation to learn diverse and rewarding skillscited by 7
- 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
- 2023Creating Multi-Level Skill Hierarchies in Reinforcement Learningcited by 5
- 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
- 2023Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.