# Hierarchical Reinforcement Learning for Quadruped Locomotion

arXiv: https://arxiv.org/abs/1905.08926 · HTML: https://mlsanity.com/paper/1905.08926/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2019.

## Built on by (most cited first, 24 of 24)

- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots](https://mlsanity.com/paper/2405.01792/) (2024)
- [Neural Volumetric Memory for Visual Locomotion Control](https://mlsanity.com/paper/2304.01201/) (2023)
- [DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2210.05150/) (2022)
- [PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations](https://mlsanity.com/paper/2207.13224/) (2022)
- [Cascaded Compositional Residual Learning for Complex Interactive Behaviors](https://mlsanity.com/paper/2212.08954/) (2022)
- [Learning to Centralize Dual-Arm Assembly](https://mlsanity.com/paper/2110.04003/) (2021)
- [Hierarchical generative modelling for autonomous robots](https://mlsanity.com/paper/2308.07775/) (2023)
- [Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environments](https://mlsanity.com/paper/2209.09233/) (2022)
- [Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space](https://mlsanity.com/paper/2110.09904/) (2021)
- [Unlocking Pixels for Reinforcement Learning via Implicit Attention](https://mlsanity.com/paper/2102.04353/) (2021)
- [End-to-End Reinforcement Learning for Torque Based Variable Height Hopping](https://mlsanity.com/paper/2307.16676/) (2023)
- [LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba](https://mlsanity.com/paper/2508.11849/) (2025)
- [QuadKAN: KAN-Enhanced Quadruped Motion Control via End-to-End Reinforcement Learning](https://mlsanity.com/paper/2508.19153/) (2025)
- [Behavior evolution-inspired approach to walking gait reinforcement training for quadruped robots](https://mlsanity.com/paper/2409.16862/) (2024)
- [SATA: Safe and Adaptive Torque-Based Locomotion Policies Inspired by Animal Learning](https://mlsanity.com/paper/2502.12674/) (2025)
- [HEIR: Learning Graph-Based Motion Hierarchies](https://mlsanity.com/paper/2510.26786/) (2025)
- [SHIRO: Soft Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2212.12786/) (2022)
- [Unlock Reliable Skill Inference for Quadruped Adaptive Behavior by Skill Graph](https://mlsanity.com/paper/2311.06015/) (2023)
- [Locomotion Generation for a Rat Robot based on Environmental Changes via Reinforcement Learning](https://mlsanity.com/paper/2403.11788/) (2024)
- [Dynamic Legged Ball Manipulation on Rugged Terrains with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2504.14989/) (2025)
- [Quadrupedal Robot Skateboard Mounting via Reverse Curriculum Learning](https://mlsanity.com/paper/2505.06561/) (2025)
- [Vision-Proprioception Fusion with Mamba2 in End-to-End Reinforcement Learning for Motion Control](https://mlsanity.com/paper/2509.07593/) (2025)
- [Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped](https://mlsanity.com/paper/2610.05855/) (2026)

## Abstract (arXiv)

Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hierarchical framework to automatically decompose complex locomotion tasks. A high-level policy issues commands in a latent space and also selects for how long the low-level policy will execute the latent command. Concurrently, the low-level policy uses the latent command and only the robot's on-board sensors to control the robot's actuators. Our approach allows the high-level policy to run at a lower frequency than the low-level one. We test our framework on a path-following task for a dynamic quadruped robot and we show that steering behaviors automatically emerge in the latent command space as low-level skills are needed for this task. We then show efficient adaptation of the trained policy to a different task by transfer of the trained low-level policy. Finally, we validate the policies on a real quadruped robot. To the best of our knowledge, this is the first application of end-to-end hierarchical learning to a real robotic locomotion task.