# Composing Task-Agnostic Policies with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.10681 · HTML: https://mlsanity.com/paper/1905.10681/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 20 of 20)

- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [Controllability-Aware Unsupervised Skill Discovery](https://mlsanity.com/paper/2302.05103/) (2023)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Skill Expansion and Composition in Parameter Space](https://mlsanity.com/paper/2502.05932/) (2025)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [ASPiRe:Adaptive Skill Priors for Reinforcement Learning](https://mlsanity.com/paper/2209.15205/) (2022)
- [Toward Robust Long Range Policy Transfer](https://mlsanity.com/paper/2103.02957/) (2021)
- [Control Transformer: Robot Navigation in Unknown Environments through PRM-Guided Return-Conditioned Sequence Modeling](https://mlsanity.com/paper/2211.06407/) (2022)
- [Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery](https://mlsanity.com/paper/2204.13906/) (2022)
- [Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2210.03729/) (2022)
- [Hierarchical Primitive Composition: Simultaneous Activation of Skills with Inconsistent Action Dimensions in Multiple Hierarchies](https://mlsanity.com/paper/2110.01833/) (2021)
- [Learning Multi-Objective Curricula for Robotic Policy Learning](https://mlsanity.com/paper/2110.03032/) (2021)
- [A Constrained-Optimization Approach to the Execution of Prioritized Stacks of Learned Multi-Robot Tasks](https://mlsanity.com/paper/2301.05346/) (2023)
- [On the Value of Myopic Behavior in Policy Reuse](https://mlsanity.com/paper/2305.17623/) (2023)
- [Learning Parameterized Skills from Demonstrations](https://mlsanity.com/paper/2510.24095/) (2025)
- [Policy composition in reinforcement learning via multi-objective policy optimization](https://mlsanity.com/paper/2308.15470/) (2023)
- [Multimodal LLMs Do Not Compose Skills Optimally Across Modalities](https://mlsanity.com/paper/2511.08113/) (2025)

## Abstract (arXiv)

The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific policies or skills but almost no focus on composing necessary, task-agnostic skills to find a solution to new problems. In this paper, we propose a novel deep reinforcement learning-based skill transfer and composition method that takes the agent's primitive policies to solve unseen tasks. We evaluate our method in difficult cases where training policy through standard reinforcement learning (RL) or even hierarchical RL is either not feasible or exhibits high sample complexity. We show that our method not only transfers skills to new problem settings but also solves the challenging environments requiring both task planning and motion control with high data efficiency.