# Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies

arXiv: https://arxiv.org/abs/2001.00248 · HTML: https://mlsanity.com/paper/2001.00248/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2020.

## Built on by (most cited first, 25 of 25)

- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos](https://mlsanity.com/paper/2406.01486/) (2024)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Multimodal Subtask Graph Generation from Instructional Videos](https://mlsanity.com/paper/2302.08672/) (2023)
- [SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation](https://mlsanity.com/paper/2410.18065/) (2024)
- [Multi-Task Learning with Sequence-Conditioned Transporter Networks](https://mlsanity.com/paper/2109.07578/) (2021)
- [Composing Task Knowledge with Modular Successor Feature Approximators](https://mlsanity.com/paper/2301.12305/) (2023)
- [Unsupervised Task Graph Generation from Instructional Video Transcripts](https://mlsanity.com/paper/2302.09173/) (2023)
- [Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2307.03486/) (2023)
- [Possibility Before Utility: Learning And Using Hierarchical Affordances](https://mlsanity.com/paper/2203.12686/) (2022)
- [TOD-Flow: Modeling the Structure of Task-Oriented Dialogues](https://mlsanity.com/paper/2312.04668/) (2023)
- [Learning Parameterized Task Structure for Generalization to Unseen Entities](https://mlsanity.com/paper/2203.15034/) (2022)
- [Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization](https://mlsanity.com/paper/2205.12648/) (2022)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Code Models are Zero-shot Precondition Reasoners](https://mlsanity.com/paper/2311.09601/) (2023)
- [Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division](https://mlsanity.com/paper/2203.14855/) (2022)
- [Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videos](https://mlsanity.com/paper/2502.17753/) (2025)
- [Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward](https://mlsanity.com/paper/2305.00508/) (2023)
- [ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videos](https://mlsanity.com/paper/2512.03479/) (2025)
- [ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistency](https://mlsanity.com/paper/2606.25317/) (2026)
- [Feature-Attending Recurrent Modules for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2112.08369/) (2021)
- [Matching options to tasks using Option-Indexed Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2206.05750/) (2022)
- [Concept Discovery for Fast Adapatation](https://mlsanity.com/paper/2301.07850/) (2023)
- [Boosting Hierarchical Reinforcement Learning with Meta-Learning for Complex Task Adaptation](https://mlsanity.com/paper/2410.07921/) (2024)

## Abstract (arXiv)

We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph which describes a set of subtasks and their dependencies that are unknown to the agent. The agent needs to quickly adapt to the task over few episodes during adaptation phase to maximize the return in the test phase. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference(MSGI), which infers the latent parameter of the task by interacting with the environment and maximizes the return given the latent parameter. To facilitate learning, we adopt an intrinsic reward inspired by upper confidence bound (UCB) that encourages efficient exploration. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter, and to adapt more efficiently than existing meta RL and hierarchical RL methods.