# Integrating Model-Based Footstep Planning with Model-Free Reinforcement Learning for Dynamic Legged Locomotion

arXiv: https://arxiv.org/abs/2408.02662 · HTML: https://mlsanity.com/paper/2408.02662/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Perceptive Locomotion through Nonlinear Model Predictive Control](https://mlsanity.com/paper/2208.08373/) (2022)
- [Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model](https://mlsanity.com/paper/2010.11234/) (2020)
- [Sim-to-Real Learning of Footstep-Constrained Bipedal Dynamic Walking](https://mlsanity.com/paper/2203.07589/) (2022)
- [Learning Dynamic Bipedal Walking Across Stepping Stones](https://mlsanity.com/paper/2205.01807/) (2022)
- [Optimizing Bipedal Maneuvers of Single Rigid-Body Models for Reinforcement Learning](https://mlsanity.com/paper/2207.04163/) (2022)

## Built on by (most cited first, 10 of 10)

- [CusADi: A GPU Parallelization Framework for Symbolic Expressions and Optimal Control](https://mlsanity.com/paper/2408.09662/) (2024)
- [CLF-RL: Control Lyapunov Function Guided Reinforcement Learning](https://mlsanity.com/paper/2508.09354/) (2025)
- [Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning](https://mlsanity.com/paper/2509.19573/) (2025)
- [Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control](https://mlsanity.com/paper/2510.12717/) (2025)
- [LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots](https://mlsanity.com/paper/2509.09106/) (2025)
- [Physics-Guided Residual Reinforcement Learning for Humanoid Narrow-Path Traversal](https://mlsanity.com/paper/2508.20661/) (2025)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)
- [Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking](https://mlsanity.com/paper/2606.08253/) (2026)
- [ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion](https://mlsanity.com/paper/2607.03454/) (2026)
- [Terrain Consistent Reference-Guided RL for Humanoid Navigation Autonomy](https://mlsanity.com/paper/2605.15517/) (2026)

## Abstract (arXiv)

In this work, we introduce a control framework that combines model-based footstep planning with Reinforcement Learning (RL), leveraging desired footstep patterns derived from the Linear Inverted Pendulum (LIP) dynamics. Utilizing the LIP model, our method forward predicts robot states and determines the desired foot placement given the velocity commands. We then train an RL policy to track the foot placements without following the full reference motions derived from the LIP model. This partial guidance from the physics model allows the RL policy to integrate the predictive capabilities of the physics-informed dynamics and the adaptability characteristics of the RL controller without overfitting the policy to the template model. Our approach is validated on the MIT Humanoid, demonstrating that our policy can achieve stable yet dynamic locomotion for walking and turning. We further validate the adaptability and generalizability of our policy by extending the locomotion task to unseen, uneven terrain. During the hardware deployment, we have achieved forward walking speeds of up to 1.5 m/s on a treadmill and have successfully performed dynamic locomotion maneuvers such as 90-degree and 180-degree turns.