mlsanityData to 7 Oct 2026

Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies

arXiv 1807.07665 ↗First posted 19 Jul 2018cs.LG cs.AI stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

We introduce a new RL problem where the agent is required to generalize to a previously-unseen environment characterized by a subtask graph which describes a set of subtasks and their dependencies. Unlike existing hierarchical multitask RL approaches that explicitly describe what the agent should do at a high level, our problem only describes properties of subtasks and relationships among them, which requires the agent to perform complex reasoning to find the optimal subtask to execute. To solve this problem, we propose a neural subtask graph solver (NSGS) which encodes the subtask graph using a recursive neural network embedding. To overcome the difficulty of training, we propose a novel non-parametric gradient-based policy, graph reward propagation, to pre-train our NSGS agent and further finetune it through actor-critic method. The experimental results on two 2D visual domains show that our agent can perform complex reasoning to find a near-optimal way of executing the subtask graph and generalize well to the unseen subtask graphs. In addition, we compare our agent with a Monte-Carlo tree search (MCTS) method showing that our method is much more efficient than MCTS, and the performance of NSGS can be further improved by combining it with MCTS.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 11; 2023: 10; 2024: 2; 2025: 3; 2026: 7.

43 papers.

Sort:

By area:

  1. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  2. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  3. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  4. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  5. 2023A Survey on Causal Reinforcement Learningcited by 43
  6. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  7. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 23
  8. 2022Intelligent problem-solving as integrated hierarchical reinforcement learningcited by 21
  9. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  10. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19
  11. 2023Multimodal Subtask Graph Generation from Instructional Videoscited by 13
  12. 2021Generalization in Text-based Games via Hierarchical Reinforcement Learningcited by 11
31 more
  1. 2022Perceiving the World: Question-guided Reinforcement Learning for Text-based Gamescited by 10
  2. 2021Multi-Task Learning with Sequence-Conditioned Transporter Networkscited by 8
  3. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 8
  4. 2021Program Synthesis Guided Reinforcement Learning for Partially Observed Environmentscited by 7
  5. 2022One-Shot Learning from a Demonstration with Hierarchical Latent Languagecited by 6
  6. 2023A Picture is Worth a Thousand Words: Language Models Plan from Pixelscited by 6
  7. 2021Reinforcement Learning of Implicit and Explicit Control Flow in Instructionscited by 4
  8. 2023Robust Subtask Learning for Compositional Generalizationcited by 4
  9. 2023TOD-Flow: Modeling the Structure of Task-Oriented Dialoguescited by 4
  10. 2022Learning Parameterized Task Structure for Generalization to Unseen Entitiescited by 3
  11. 2022Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalizationcited by 3
  12. 2023Code Models are Zero-shot Precondition Reasonerscited by 3
  13. 2024Inductive Generalization in Reinforcement Learning from Specificationscited by 3
  14. 2025Filtering Learning Histories Enhances In-Context Reinforcement Learningcited by 3
  15. 2022Goal-Conditioned Q-Learning as Knowledge Distillationcited by 1
  16. 2022Generalizing LTL Instructions via Future Dependent Optionscited by 1
  17. 2023Dyadic Reinforcement Learningcited by 1
  18. 2023Progressively Efficient Learningcited by 1
  19. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 1
  20. 2025D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detectioncited by 1
  21. 2026Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalizationcited by 1
  22. 2026Zero-shot Imitation Learning by Latent Topology Mappingcited by 1
  23. 2021Feature-Attending Recurrent Modules for Generalization in Reinforcement Learningcited by 0
  24. 2022Learning Invariable Semantical Representation from Language for Extensible Policy Generalizationcited by 0
  25. 2023Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Makingcited by 0
  26. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventionscited by 0
  27. 2026Delay-Empowered Causal Hierarchical Reinforcement Learningcited by 0
  28. 2026Decoupled Behavioral Cloning for Scalable Inductive Generalization in RL from Specificationscited by 0
  29. 2026Inductive Generalization for Robotic Manipulationcited by 0
  30. 2026Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agentscited by 0
  31. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0