# Toward Robust Long Range Policy Transfer

arXiv: https://arxiv.org/abs/2103.02957 · HTML: https://mlsanity.com/paper/2103.02957/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies](https://mlsanity.com/paper/1905.09808/) (2019)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [ProMP: Proximal Meta-Policy Search](https://mlsanity.com/paper/1810.06784/) (2018)
- [InfoBot: Transfer and Exploration via the Information Bottleneck](https://mlsanity.com/paper/1901.10902/) (2019)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Latent Space Policies for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1804.02808/) (2018)
- [Information asymmetry in KL-regularized RL](https://mlsanity.com/paper/1905.01240/) (2019)
- [Sub-policy Adaptation for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1906.05862/) (2019)
- [Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real](https://mlsanity.com/paper/1908.05224/) (2019)
- [Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards](https://mlsanity.com/paper/1910.04450/) (2019)
- [Variance Reduction for Reinforcement Learning in Input-Driven Environments](https://mlsanity.com/paper/1807.02264/) (2018)
- [Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives](https://mlsanity.com/paper/1906.10667/) (2019)
- [Hierarchical Reinforcement Learning with Hindsight](https://mlsanity.com/paper/1805.08180/) (2018)
- [Composing Task-Agnostic Policies with Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10681/) (2019)

## Built on by (most cited first, 6 of 6)

- [CLA-NeRF: Category-Level Articulated Neural Radiance Field](https://mlsanity.com/paper/2202.00181/) (2022)
- [Self-Composing Policies for Scalable Continual Reinforcement Learning](https://mlsanity.com/paper/2506.14811/) (2025)
- [ASPiRe:Adaptive Skill Priors for Reinforcement Learning](https://mlsanity.com/paper/2209.15205/) (2022)
- [Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2210.03729/) (2022)
- [Meta-CPR: Generalize to Unseen Large Number of Agents with Communication Pattern Recognition Module](https://mlsanity.com/paper/2112.07222/) (2021)
- [Incomplete Knowledge Graph Alignment](https://mlsanity.com/paper/2112.09266/) (2021)

## Abstract (arXiv)

Humans can master a new task within a few trials by drawing upon skills acquired through prior experience. To mimic this capability, hierarchical models combining primitive policies learned from prior tasks have been proposed. However, these methods fall short comparing to the human's range of transferability. We propose a method, which leverages the hierarchical structure to train the combination function and adapt the set of diverse primitive polices alternatively, to efficiently produce a range of complex behaviors on challenging new tasks. We also design two regularization terms to improve the diversity and utilization rate of the primitives in the pre-training phase. We demonstrate that our method outperforms other recent policy transfer methods by combining and adapting these reusable primitives in tasks with continuous action space. The experiment results further show that our approach provides a broader transferring range. The ablation study also shows the regularization terms are critical for long range policy transfer. Finally, we show that our method consistently outperforms other methods when the quality of the primitives varies.