# Learning from Trajectories via Subgoal Discovery

arXiv: https://arxiv.org/abs/1911.07224 · HTML: https://mlsanity.com/paper/1911.07224/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2019.

## Built on by (most cited first, 28 of 28)

- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Cross-domain Imitation from Observations](https://mlsanity.com/paper/2105.10037/) (2021)
- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [Subgoal Search For Complex Reasoning Tasks](https://mlsanity.com/paper/2108.11204/) (2021)
- [Hierarchical Decision Transformer](https://mlsanity.com/paper/2209.10447/) (2022)
- [Hierarchical Imitation Learning with Vector Quantized Models](https://mlsanity.com/paper/2301.12962/) (2023)
- [A Survey of Demonstration Learning](https://mlsanity.com/paper/2303.11191/) (2023)
- [Efficient Planning with Latent Diffusion](https://mlsanity.com/paper/2310.00311/) (2023)
- [Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2104.06411/) (2021)
- [Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning](https://mlsanity.com/paper/2601.18984/) (2026)
- [Learning World Models for Unconstrained Goal Navigation](https://mlsanity.com/paper/2411.02446/) (2024)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Learning Rational Subgoals from Demonstrations and Instructions](https://mlsanity.com/paper/2303.05487/) (2023)
- [Curricular Subgoals for Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.08232/) (2023)
- [BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping](https://mlsanity.com/paper/2409.05358/) (2024)
- [Identifying Selections for Unsupervised Subtask Discovery](https://mlsanity.com/paper/2410.21616/) (2024)
- [Leveraging Demonstrations with Latent Space Priors](https://mlsanity.com/paper/2210.14685/) (2022)
- [Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/2602.08557/) (2026)
- [D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioning](https://mlsanity.com/paper/2210.14428/) (2022)
- [Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance](https://mlsanity.com/paper/2409.03996/) (2024)
- [Generalizing to New Tasks via One-Shot Compositional Subgoals](https://mlsanity.com/paper/2205.07716/) (2022)
- [Strategy Extraction in Single-Agent Games](https://mlsanity.com/paper/2305.12623/) (2023)
- [Reward Shaping with Dynamic Trajectory Aggregation](https://mlsanity.com/paper/2104.06163/) (2021)
- [Reward Shaping with Subgoals for Social Navigation](https://mlsanity.com/paper/2104.06410/) (2021)
- [Multi-Task Option Learning and Discovery for Stochastic Path Planning](https://mlsanity.com/paper/2210.00068/) (2022)
- [Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies](https://mlsanity.com/paper/2406.18053/) (2024)
- [RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation](https://mlsanity.com/paper/2603.15469/) (2026)
- [Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation](https://mlsanity.com/paper/2609.19414/) (2026)

## Abstract (arXiv)

Learning to solve complex goal-oriented tasks with sparse terminal-only rewards often requires an enormous number of samples. In such cases, using a set of expert trajectories could help to learn faster. However, Imitation Learning (IL) via supervised pre-training with these trajectories may not perform as well and generally requires additional finetuning with expert-in-the-loop. In this paper, we propose an approach which uses the expert trajectories and learns to decompose the complex main task into smaller sub-goals. We learn a function which partitions the state-space into sub-goals, which can then be used to design an extrinsic reward function. We follow a strategy where the agent first learns from the trajectories using IL and then switches to Reinforcement Learning (RL) using the identified sub-goals, to alleviate the errors in the IL step. To deal with states which are under-represented by the trajectory set, we also learn a function to modulate the sub-goal predictions. We show that our method is able to solve complex goal-oriented tasks, which other RL, IL or their combinations in literature are not able to solve.