# Learning Rational Subgoals from Demonstrations and Instructions

arXiv: https://arxiv.org/abs/2303.05487 · HTML: https://mlsanity.com/paper/2303.05487/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Neural Logic Machines](https://mlsanity.com/paper/1904.11694/) (2019)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning](https://mlsanity.com/paper/2011.00517/) (2020)
- [Learning from Trajectories via Subgoal Discovery](https://mlsanity.com/paper/1911.07224/) (2019)
- [Alignment-based compositional semantics for instruction following](https://mlsanity.com/paper/1508.06491/) (2015)
- [Learning Task Decomposition with Ordered Memory Policy Network](https://mlsanity.com/paper/2103.10972/) (2021)
- [Unsupervised Learning of Temporal Abstractions with Slot-based Transformers](https://mlsanity.com/paper/2203.13573/) (2022)
- [Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization](https://mlsanity.com/paper/2203.02214/) (2022)
- [Subgoal Discovery for Hierarchical Dialogue Policy Learning](https://mlsanity.com/paper/1804.07855/) (2018)
- [Learning and Planning for Temporally Extended Tasks in Unknown Environments](https://mlsanity.com/paper/2104.10636/) (2021)
- [Planning with Abstract Learned Models While Learning Transferable Subtasks](https://mlsanity.com/paper/1912.07544/) (2019)

## Built on by (most cited first, 5 of 5)

- [Learning adaptive planning representations with natural language guidance](https://mlsanity.com/paper/2312.08566/) (2023)
- [Grounding Language Plans in Demonstrations Through Counterfactual Perturbations](https://mlsanity.com/paper/2403.17124/) (2024)
- [Learning Planning Abstractions from Language](https://mlsanity.com/paper/2405.03864/) (2024)
- [Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance](https://mlsanity.com/paper/2409.03996/) (2024)
- [Learning Hidden Subgoals under Temporal Ordering Constraints in Reinforcement Learning](https://mlsanity.com/paper/2411.01425/) (2024)

## Abstract (arXiv)

We present a framework for learning useful subgoals that support efficient long-term planning to achieve novel goals. At the core of our framework is a collection of rational subgoals (RSGs), which are essentially binary classifiers over the environmental states. RSGs can be learned from weakly-annotated data, in the form of unsegmented demonstration trajectories, paired with abstract task descriptions, which are composed of terms initially unknown to the agent (e.g., collect-wood then craft-boat then go-across-river). Our framework also discovers dependencies between RSGs, e.g., the task collect-wood is a helpful subgoal for the task craft-boat. Given a goal description, the learned subgoals and the derived dependencies facilitate off-the-shelf planning algorithms, such as A* and RRT, by setting helpful subgoals as waypoints to the planner, which significantly improves performance-time efficiency.