# Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies

arXiv: https://arxiv.org/abs/2112.05062 · HTML: https://mlsanity.com/paper/2112.05062/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 30 of 35)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning](https://mlsanity.com/paper/2010.13611/) (2020)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [Demonstration-Guided Reinforcement Learning with Learned Skills](https://mlsanity.com/paper/2107.10253/) (2021)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [Latent Space Policies for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1804.02808/) (2018)
- [Task-Embedded Control Networks for Few-Shot Imitation Learning](https://mlsanity.com/paper/1810.03237/) (2018)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [TACO: Learning Task Decomposition via Temporal Alignment for Control](https://mlsanity.com/paper/1803.01840/) (2018)
- [Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors](https://mlsanity.com/paper/2006.13205/) (2020)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Information asymmetry in KL-regularized RL](https://mlsanity.com/paper/1905.01240/) (2019)
- [Data-efficient Hindsight Off-policy Option Learning](https://mlsanity.com/paper/2007.15588/) (2020)

## Built on by (most cited first, 26 of 26)

- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [Diffusion Reward: Learning Rewards via Conditional Video Diffusion](https://mlsanity.com/paper/2312.14134/) (2023)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors](https://mlsanity.com/paper/2305.04412/) (2023)
- [SMPLOlympics: Sports Environments for Physically Simulated Humanoids](https://mlsanity.com/paper/2407.00187/) (2024)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [Efficient Planning with Latent Diffusion](https://mlsanity.com/paper/2310.00311/) (2023)
- [Language-guided Skill Learning with Temporal Variational Inference](https://mlsanity.com/paper/2402.16354/) (2024)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [ASPiRe:Adaptive Skill Priors for Reinforcement Learning](https://mlsanity.com/paper/2209.15205/) (2022)
- [DMAP: a Distributed Morphological Attention Policy for Learning to Locomote with a Changing Body](https://mlsanity.com/paper/2209.14218/) (2022)
- [SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration](https://mlsanity.com/paper/2211.13743/) (2022)
- [Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domains](https://mlsanity.com/paper/2302.12617/) (2023)
- [Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning](https://mlsanity.com/paper/2201.08115/) (2022)
- [Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysis](https://mlsanity.com/paper/2206.09046/) (2022)
- [Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning](https://mlsanity.com/paper/2509.21942/) (2025)
- [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](https://mlsanity.com/paper/2411.07760/) (2024)
- [Boosting Reinforcement Learning and Planning with Demonstrations: A Survey](https://mlsanity.com/paper/2303.13489/) (2023)
- [Learning Multimodal Latent Dynamics for Human-Robot Interaction](https://mlsanity.com/paper/2311.16380/) (2023)
- [Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach](https://mlsanity.com/paper/2505.06482/) (2025)
- [HiBerNAC: Hierarchical Brain-inspired Robotic Neural Agent Collective for Disentangling Complex Manipulation](https://mlsanity.com/paper/2506.08296/) (2025)
- [Learning Parameterized Skills from Demonstrations](https://mlsanity.com/paper/2510.24095/) (2025)
- [Unlock Reliable Skill Inference for Quadruped Adaptive Behavior by Skill Graph](https://mlsanity.com/paper/2311.06015/) (2023)
- [On the benefits of pixel-based hierarchical policies for task generalization](https://mlsanity.com/paper/2407.19142/) (2024)
- [Data Augmentation for Instruction Following Policies via Trajectory Segmentation](https://mlsanity.com/paper/2503.01871/) (2025)

## Abstract (arXiv)

For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model. In contrast to existing work, our method exploits a three-level hierarchy of both discrete and continuous latent variables, to capture a set of high-level behaviours while allowing for variance in how they are executed. We demonstrate in manipulation domains that the method can effectively cluster offline data into distinct, executable behaviours, while retaining the flexibility of a continuous latent variable model. The resulting skills can be transferred and fine-tuned on new tasks, unseen objects, and from state to vision-based policies, yielding better sample efficiency and asymptotic performance compared to existing skill- and imitation-based methods. We further analyse how and when the skills are most beneficial: they encourage directed exploration to cover large regions of the state space relevant to the task, making them most effective in challenging sparse-reward settings.