Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion
On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation
Abstract (arXiv)
We propose a novel hierarchical reinforcement learning framework for quadruped locomotion over challenging terrain. Our approach incorporates a two-layer hierarchy in which a high-level policy (HLP) selects optimal goals for a low-level policy (LLP). The LLP is trained using an on-policy actor-critic RL algorithm and is given footstep placements as goals. We propose an HLP that does not require any additional training or environment samples and instead operates via an online optimization process over the learned value function of the LLP. We demonstrate the benefits of this framework by comparing it with an end-to-end reinforcement learning (RL) approach. We observe improvements in its ability to achieve higher rewards with fewer collisions across an array of different terrains, including terrains more difficult than any encountered during training.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 22 references that are in the map. First posted on arXiv 24 Jun 2025.
Builds on
20 of 22 references.
References · 20
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
- 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
- 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
- 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
- 2018Learning to Walk via Deep Reinforcement Learningcited by 158
- 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
- 2020Learning to Walk in the Real World with Minimal Human Effortcited by 111
- 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 79
- 2020ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skillscited by 54
8 more
- 2019DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learningcited by 50
- 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 50
- 2019Policies Modulating Trajectory Generatorscited by 49
- 2019Learning Fast Adaptation with Meta Strategy Optimizationcited by 48
- 2020Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotioncited by 25
- 2019Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learningcited by 13
- 2020Zero-Shot Terrain Generalization for Visual Locomotion Policiescited by 13
- 2020Learning Agile Locomotion Skills with a Mentorcited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Motion Priors Reimagined: Adapting Flat-Terrain Skills for Complex Quadruped Mobility
- 2026Task-Level Decisions to Gait Level Control: A Hierarchical Policy Approach for Quadruped Navigation ↗
- 2025Learning Terrain-Specialized Policies for Adaptive Locomotion in Challenging Environments
- 2026Energy-Efficient Gait Adaptation via Hierarchical Reinforcement Learning for Quadrupedal Locomotion Across Diverse Terrains
- 2021Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner
- 2025Dynamic Legged Ball Manipulation on Rugged Terrains with Hierarchical Reinforcement Learning
- 2026Towards Torque-Driven Reinforcement Learning for Quadruped Locomotion
- 2025Acrobotics: A Generalist Approach to Quadrupedal Robots' Parkour