# Identifying Selections for Unsupervised Subtask Discovery

arXiv: https://arxiv.org/abs/2410.21616 · HTML: https://mlsanity.com/paper/2410.21616/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Causal Inference in the Presence of Latent Variables and Selection Bias](https://mlsanity.com/paper/1302.4983/) (2013)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2109.13841/) (2021)
- [Multi-Level Discovery of Deep Options](https://mlsanity.com/paper/1703.08294/) (2017)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations](https://mlsanity.com/paper/1710.05421/) (2017)
- [Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information](https://mlsanity.com/paper/1810.01266/) (2018)
- [Variational Temporal Abstraction](https://mlsanity.com/paper/1910.00775/) (2019)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Learning from Trajectories via Subgoal Discovery](https://mlsanity.com/paper/1911.07224/) (2019)
- [SKID RAW: Skill Discovery from Raw Trajectories](https://mlsanity.com/paper/2103.14610/) (2021)
- [Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policies](https://mlsanity.com/paper/2103.16772/) (2021)
- [Adversarial Option-Aware Hierarchical Imitation Learning](https://mlsanity.com/paper/2106.05530/) (2021)
- [Causal Calculus in the Presence of Cycles, Latent Confounders and Selection Bias](https://mlsanity.com/paper/1901.00433/) (2019)
- [Local Constraint-Based Causal Discovery under Selection Bias](https://mlsanity.com/paper/2203.01848/) (2022)
- [Multi-task Hierarchical Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2305.12633/) (2023)
- [Programmable Agents](https://mlsanity.com/paper/1706.06383/) (2017)
- [Detecting and Identifying Selection Structure in Sequential Data](https://mlsanity.com/paper/2407.00529/) (2024)
- [Foundations of Structural Causal Models with Latent Selection](https://mlsanity.com/paper/2401.06925/) (2024)
- [PODNet: A Neural Network for Discovery of Plannable Options](https://mlsanity.com/paper/1911.00171/) (2019)

## Built on by (most cited first, 4 of 4)

- [Latent Variable Causal Discovery under Selection Bias](https://mlsanity.com/paper/2512.11219/) (2025)
- [From Generalist to Specialist Representation](https://mlsanity.com/paper/2605.12733/) (2026)
- [Towards a holistic understanding of Selection Bias for Causal Effect Identification](https://mlsanity.com/paper/2605.13430/) (2026)
- [Causal Modeling of Selection in Evolution](https://mlsanity.com/paper/2606.05689/) (2026)

## Abstract (arXiv)

When solving long-horizon tasks, it is intriguing to decompose the high-level task into subtasks. Decomposing experiences into reusable subtasks can improve data efficiency, accelerate policy generalization, and in general provide promising solutions to multi-task reinforcement learning and imitation learning problems. However, the concept of subtasks is not sufficiently understood and modeled yet, and existing works often overlook the true structure of the data generation process: subtasks are the results of a textitselection mechanism on actions, rather than possible underlying confounders or intermediates. Specifically, we provide a theory to identify, and experiments to verify the existence of selection variables in such data. These selections serve as subgoals that indicate subtasks and guide policy. In light of this idea, we develop a sequential non-negative matrix factorization (seq- NMF) method to learn these subgoals and extract meaningful behavior patterns as subtasks. Our empirical results on a challenging Kitchen environment demonstrate that the learned subtasks effectively enhance the generalization to new tasks in multi-task imitation learning scenarios. The codes are provided at https://anonymous.4open.science/r/IdentifyingSelectionsforUnsupervisedSubtaskDiscovery/README.md.