# Learning Bipedal Walking On Planned Footsteps For Humanoid Robots

arXiv: https://arxiv.org/abs/2207.12644 · HTML: https://mlsanity.com/paper/2207.12644/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition](https://mlsanity.com/paper/2011.01387/) (2020)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [Learning Symmetric and Low-energy Locomotion](https://mlsanity.com/paper/1801.08093/) (2018)
- [ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills](https://mlsanity.com/paper/2005.04323/) (2020)
- [DeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning](https://mlsanity.com/paper/2106.00534/) (2021)
- [Robust Feedback Motion Policy Design Using Reinforcement Learning on a 3D Digit Bipedal Robot](https://mlsanity.com/paper/2103.15309/) (2021)
- [Sim-to-Real Learning of Footstep-Constrained Bipedal Dynamic Walking](https://mlsanity.com/paper/2203.07589/) (2022)
- [Learning natural locomotion behaviors for humanoid robots using human knowledge](https://mlsanity.com/paper/2005.10195/) (2020)
- [Stair Climbing Stabilization of the HRP-4 Humanoid Robot using Whole-body Admittance Control](https://mlsanity.com/paper/1809.07073/) (2018)

## Built on by (most cited first, 12 of 12)

- [Learning Bipedal Walking for Humanoids with Current Feedback](https://mlsanity.com/paper/2303.03724/) (2023)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [Robust Humanoid Walking on Compliant and Uneven Terrain with Deep Reinforcement Learning](https://mlsanity.com/paper/2504.13619/) (2025)
- [Learning Skateboarding for Humanoid Robots through Massively Parallel Reinforcement Learning](https://mlsanity.com/paper/2409.07846/) (2024)
- [Learning Bipedal Walking for Humanoid Robots in Challenging Environments with Obstacle Avoidance](https://mlsanity.com/paper/2410.08212/) (2024)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion](https://mlsanity.com/paper/2511.06465/) (2025)
- [Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking](https://mlsanity.com/paper/2606.08253/) (2026)
- [Learning Generic and Dynamic Locomotion of Humanoids Across Discrete Terrains](https://mlsanity.com/paper/2405.17227/) (2024)
- [Success in Humanoid Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2507.18883/) (2025)
- [HuMam: Humanoid Motion Control via End-to-End Deep Reinforcement Learning with Mamba](https://mlsanity.com/paper/2509.18046/) (2025)
- [Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](https://mlsanity.com/paper/2512.12993/) (2025)
- [Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped](https://mlsanity.com/paper/2610.05855/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) based controllers for legged robots have demonstrated impressive robustness for walking in different environments for several robot platforms. To enable the application of RL policies for humanoid robots in real-world settings, it is crucial to build a system that can achieve robust walking in any direction, on 2D and 3D terrains, and be controllable by a user-command. In this paper, we tackle this problem by learning a policy to follow a given step sequence. The policy is trained with the help of a set of procedurally generated step sequences (also called footstep plans). We show that simply feeding the upcoming 2 steps to the policy is sufficient to achieve omnidirectional walking, turning in place, standing, and climbing stairs. Our method employs curriculum learning on the complexity of terrains, and circumvents the need for reference motions or pre-trained weights. We demonstrate the application of our proposed method to learn RL policies for 2 new robot platforms - HRP5P and JVRC-1 - in the MuJoCo simulation environment. The code for training and evaluation is available online.