# Meta Learning Shared Hierarchies

arXiv: https://arxiv.org/abs/1710.09767 · HTML: https://mlsanity.com/paper/1710.09767/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2017.

## Built on by (most cited first, 30 of 93)

- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Self-Adapting Language Models](https://mlsanity.com/paper/2506.10943/) (2025)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization](https://mlsanity.com/paper/2111.07999/) (2021)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery](https://mlsanity.com/paper/2210.03516/) (2022)
- [How to Train Your MAML to Excel in Few-Shot Classification](https://mlsanity.com/paper/2106.16245/) (2021)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models](https://mlsanity.com/paper/2303.06571/) (2023)
- [Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense](https://mlsanity.com/paper/2410.17351/) (2024)
- [On Efficient Reinforcement Learning for Full-length Game of StarCraft II](https://mlsanity.com/paper/2209.11553/) (2022)
- [Federated Learning and Meta Learning: Approaches, Applications, and Directions](https://mlsanity.com/paper/2210.13111/) (2022)
- [DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks](https://mlsanity.com/paper/2404.16779/) (2024)
- [Mobi-π: Mobilizing Your Robot Learning Policy](https://mlsanity.com/paper/2505.23692/) (2025)
- [Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials](https://mlsanity.com/paper/2105.00990/) (2021)
- [Context-Specific Representation Abstraction for Deep Option Learning](https://mlsanity.com/paper/2109.09876/) (2021)
- [Assistive Teaching of Motor Control Tasks to Humans](https://mlsanity.com/paper/2211.14003/) (2022)
- [Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning](https://mlsanity.com/paper/2602.11779/) (2026)
- [BR-NS: an Archive-less Approach to Novelty Search](https://mlsanity.com/paper/2104.03936/) (2021)
- [Transfering Hierarchical Structure with Dual Meta Imitation Learning](https://mlsanity.com/paper/2201.11981/) (2022)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)

## Abstract (arXiv)

We develop a metalearning approach for learning hierarchically structured policies, improving sample efficiency on unseen tasks through the use of shared primitives---policies that are executed for large numbers of timesteps. Specifically, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies. We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks. We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies. We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes. We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.