# Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped

arXiv: https://arxiv.org/abs/2610.05855 · HTML: https://mlsanity.com/paper/2610.05855/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [Socially Aware Motion Planning with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.08862/) (2017)
- [Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation](https://mlsanity.com/paper/1703.00420/) (2017)
- [Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition](https://mlsanity.com/paper/2011.01387/) (2020)
- [Goal-Driven Autonomous Exploration Through Deep Reinforcement Learning](https://mlsanity.com/paper/2103.07119/) (2021)
- [Hierarchical Reinforcement Learning for Quadruped Locomotion](https://mlsanity.com/paper/1905.08926/) (2019)
- [Learning Bipedal Walking On Planned Footsteps For Humanoid Robots](https://mlsanity.com/paper/2207.12644/) (2022)
- [Learning Bipedal Walking for Humanoid Robots in Challenging Environments with Obstacle Avoidance](https://mlsanity.com/paper/2410.08212/) (2024)
- [EmoBipedNav: Emotion-aware Social Navigation for Bipedal Robots with Deep Reinforcement Learning](https://mlsanity.com/paper/2503.12538/) (2025)

## Abstract (arXiv)

A bipedal robot cannot deviate from its path to avoid an obstacle without disturbing its balance, and this coupling is most severe on underactuated platforms such as the biped considered here, which has four actuated joints per leg and no hip or ankle roll. This paper presents a Hierarchical Reinforcement Learning (HRL) framework in which a High-Level (HL) policy observes the robot pose, 36 raycast proximity measurements, moving-obstacle states, and a receding-horizon local goal, and outputs a body-velocity command (vx, vy, ωyaw) every ten control steps, while a velocity-conditioned Low-Level (LL) policy tracks each command through PD-controlled joint targets. Both policies are trained jointly with Soft Actor-Critic (SAC). Because the converged gait is task-agnostic, it is frozen and driven by classical planners over the same command interface, yielding three controlled baselines: SAC+A*, SAC+RRT*, and SAC+APF. Across 100 evaluation trials per method in randomized PyBullet environments, the proposed method reaches the goal in 98.0% of static and 88.0% of dynamic trials, against at most 78.0% and 68.0% for the planner hybrids, with path lengths within 4% of the A* reference, and ablations confirm that each observation channel and reward term contributes materially to this performance.