mlsanityData to 7 Oct 2026

Identifying Selections for Unsupervised Subtask Discovery

arXiv 2410.21616 ↗First posted 28 Oct 2024cs.LG cs.AI cs.ROcited by 4

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

When solving long-horizon tasks, it is intriguing to decompose the high-level task into subtasks. Decomposing experiences into reusable subtasks can improve data efficiency, accelerate policy generalization, and in general provide promising solutions to multi-task reinforcement learning and imitation learning problems. However, the concept of subtasks is not sufficiently understood and modeled yet, and existing works often overlook the true structure of the data generation process: subtasks are the results of a textitselection mechanism on actions, rather than possible underlying confounders or intermediates. Specifically, we provide a theory to identify, and experiments to verify the existence of selection variables in such data. These selections serve as subgoals that indicate subtasks and guide policy. In light of this idea, we develop a sequential non-negative matrix factorization (seq- NMF) method to learn these subgoals and extract meaningful behavior patterns as subtasks. Our empirical results on a challenging Kitchen environment demonstrate that the learned subtasks effectively enhance the generalization to new tasks in multi-task imitation learning scenarios. The codes are provided at https://anonymous.4open.science/r/IdentifyingSelectionsforUnsupervisedSubtaskDiscovery/README.md.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 65 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

28 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learning22 May
  2. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables20 Sep

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  7. 2019Causal Confusion in Imitation Learningcited by 286
  8. 2013Causal Inference in the Presence of Latent Variables and Selection Biascited by 165
  9. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  10. 2022Deep Hierarchical Planning from Pixelscited by 96
  11. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  12. 2017Multi-Level Discovery of Deep Optionscited by 50
14 more
  1. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  2. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Informationcited by 41
  3. 2019Variational Temporal Abstractioncited by 37
  4. 2021Hierarchical Skills for Efficient Explorationcited by 37
  5. 2019Learning from Trajectories via Subgoal Discoverycited by 27
  6. 2021SKID RAW: Skill Discovery from Raw Trajectoriescited by 22
  7. 2021Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policiescited by 20
  8. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  9. 2019Causal Calculus in the Presence of Cycles, Latent Confounders and Selection Biascited by 16
  10. 2022Local Constraint-Based Causal Discovery under Selection Biascited by 15
  11. 2017Programmable Agentscited by 14
  12. 2024Detecting and Identifying Selection Structure in Sequential Datacited by 9
  13. 2019PODNet: A Neural Network for Discovery of Plannable Optionscited by 3
  14. 2024Foundations of Structural Causal Models with Latent Selectioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Latent Variable Causal Discovery under Selection Biascited by 6
  2. 2026From Generalist to Specialist Representationcited by 0
  3. 2026Towards a holistic understanding of Selection Bias for Causal Effect Identificationcited by 0
  4. 2026Causal Modeling of Selection in Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning with Decomposed Subtasks
  2. 2025Generalisation in Multitask Fitted Q-Iteration and Offline Q-learning
  3. 2026Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning
  4. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learning
  5. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learning
  6. 2021Learning Task Decomposition with Ordered Memory Policy Network
  7. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  8. 2025Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition