# Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/1909.12324 · HTML: https://mlsanity.com/paper/1909.12324/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 13 of 13)

- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [Learning Torque Control for Quadrupedal Locomotion](https://mlsanity.com/paper/2203.05194/) (2022)
- [Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control](https://mlsanity.com/paper/2108.00128/) (2021)
- [Cascaded Compositional Residual Learning for Complex Interactive Behaviors](https://mlsanity.com/paper/2212.08954/) (2022)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [An Adaptable Approach to Learn Realistic Legged Locomotion without Examples](https://mlsanity.com/paper/2110.14998/) (2021)
- [A Simple Approach to Continual Learning by Transferring Skill Parameters](https://mlsanity.com/paper/2110.10255/) (2021)
- [Open-Ended Learning Strategies for Learning Complex Locomotion Skills](https://mlsanity.com/paper/2206.06796/) (2022)
- [Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion](https://mlsanity.com/paper/2506.20036/) (2025)
- [Learning Terrain-Specialized Policies for Adaptive Locomotion in Challenging Environments](https://mlsanity.com/paper/2509.20635/) (2025)
- [Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent](https://mlsanity.com/paper/2210.08003/) (2022)
- [Conditionally Combining Robot Skills using Large Language Models](https://mlsanity.com/paper/2310.17019/) (2023)

## Abstract (arXiv)

Learning to locomote to arbitrary goals on hardware remains a challenging problem for reinforcement learning. In this paper, we present a hierarchical learning framework that improves sample-efficiency and generalizability of locomotion skills on real-world robots. Our approach divides the problem of goal-oriented locomotion into two sub-problems: learning diverse primitives skills, and using model-based planning to sequence these skills. We parametrize our primitives as cyclic movements, improving sample-efficiency of learning on a 18 degrees of freedom robot. Then, we learn coarse dynamics models over primitive cycles and use them in a model predictive control framework. This allows us to learn to walk to arbitrary goals up to 12m away, after about two hours of training from scratch on hardware. Our results on a Daisy hexapod hardware and simulation demonstrate the efficacy of our approach at reaching distant targets, in different environments and with sensory noise.