mlsanityData to 14 Oct 2026

Robust Subtask Learning for Compositional Generalization

arXiv 2302.02984 ↗First posted 6 Feb 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks. Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform each subtask. In this paper, we focus on the problem of training subtask policies in a way that they can be used to perform any task; here, a task is given by a sequence of subtasks. We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance. We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks. We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RL algorithms to our setting and the other is an asynchronous version which enables parallel training of subtask policies. We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform state-of-the-art baselines.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

26 of 52 references.

References · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  8. 2017Robust Adversarial Reinforcement Learningcited by 451
  9. 2016The Option-Critic Architecturecited by 448
  10. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  11. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  12. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
14 more
  1. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 113
  2. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  3. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  4. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  5. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
  6. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  7. 2017Online Reinforcement Learning in Stochastic Gamescited by 67
  8. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 46
  9. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  10. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  11. 2016Learning Nash Equilibrium for General-Sum Markov Games from Batch Datacited by 20
  12. 2016Generalizing Skills with Semi-Supervised Reinforcement Learningcited by 18
  13. 2020Abstract Value Iteration for Hierarchical Reinforcement Learningcited by 12
  14. 2018Natural Option Criticcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Inductive Generalization in Reinforcement Learning from Specificationscited by 3
  2. 2025State Combinatorial Generalization In Decision Making With Conditional Diffusion Modelscited by 1
  3. 2024LLM-Augmented Symbolic Reinforcement Learning with Landmark-Based Task Decompositioncited by 0
  4. 2026Towards Compositional Generalization in LLMs for Smart Contract Security: A Case Study on Reentrancy Vulnerabilitiescited by 0