# RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control

arXiv: https://arxiv.org/abs/2407.17683 · HTML: https://mlsanity.com/paper/2407.17683/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Feedback Control For Cassie With Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05580/) (2018)
- [Learning Memory-Based Control for Human-Scale Bipedal Locomotion](https://mlsanity.com/paper/2006.02402/) (2020)
- [RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion](https://mlsanity.com/paper/2305.17842/) (2023)
- [Learning Task Space Actions for Bipedal Locomotion](https://mlsanity.com/paper/2011.04741/) (2020)
- [Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model](https://mlsanity.com/paper/2010.11234/) (2020)
- [Template Model Inspired Task Space Learning for Robust Bipedal Locomotion](https://mlsanity.com/paper/2309.15442/) (2023)
- [Dynamic Locomotion For Passive-Ankle Biped Robots And Humanoids Using Whole-Body Locomotion Control](https://mlsanity.com/paper/1901.08100/) (2019)
- [Online Non-linear Centroidal MPC for Humanoid Robot Locomotion with Step Adjustment](https://mlsanity.com/paper/2203.04489/) (2022)
- [Zero Dynamics, Pendulum Models, and Angular Momentum in Feedback Control of Bipedal Locomotion](https://mlsanity.com/paper/2105.08170/) (2021)
- [Reinforcement Learning for Reduced-order Models of Legged Robots](https://mlsanity.com/paper/2310.09873/) (2023)
- [Terrain-Adaptive, ALIP-Based Bipedal Locomotion Controller via Model Predictive Control and Virtual Constraints](https://mlsanity.com/paper/2109.14862/) (2021)
- [Control of A High Performance Bipedal Robot using Viscoelastic Liquid Cooled Actuators](https://mlsanity.com/paper/1906.03811/) (2019)
- [Data-Efficient and Safe Learning for Humanoid Locomotion Aided by a Dynamic Balancing Model](https://mlsanity.com/paper/1906.03812/) (2019)
- [MPC-Based Hierarchical Task Space Control of Underactuated and Constrained Robots for Execution of Multiple Tasks](https://mlsanity.com/paper/2009.05891/) (2020)
- [Real-Time Model Predictive Control for Industrial Manipulators with Singularity-Tolerant Hierarchical Task Control](https://mlsanity.com/paper/2209.11880/) (2022)
- [Variable Inertia Model Predictive Control for Fast Bipedal Maneuvers](https://mlsanity.com/paper/2407.16811/) (2024)

## Built on by (most cited first, 5 of 5)

- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [RLPP: A Residual Method for Zero-Shot Real-World Autonomous Racing on Scaled Platforms](https://mlsanity.com/paper/2501.17311/) (2025)
- [Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control](https://mlsanity.com/paper/2510.12717/) (2025)
- [Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation](https://mlsanity.com/paper/2606.05687/) (2026)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)

## Abstract (arXiv)

This paper proposes an online bipedal footstep planning strategy that combines model predictive control (MPC) and reinforcement learning (RL) to achieve agile and robust bipedal maneuvers. While MPC-based foot placement controllers have demonstrated their effectiveness in achieving dynamic locomotion, their performance is often limited by the use of simplified models and assumptions. To address this challenge, we develop a novel foot placement controller that leverages a learned policy to bridge the gap between the use of a simplified model and the more complex full-order robot system. Specifically, our approach employs a unique combination of an ALIP-based MPC foot placement controller for sub-optimal footstep planning and the learned policy for refining footstep adjustments, enabling the resulting footstep policy to capture the robot's whole-body dynamics effectively. This integration synergizes the predictive capability of MPC with the flexibility and adaptability of RL. We validate the effectiveness of our framework through a series of experiments using the full-body humanoid robot DRACO 3. The results demonstrate significant improvements in dynamic locomotion performance, including better tracking of a wide range of walking speeds, enabling reliable turning and traversing challenging terrains while preserving the robustness and stability of the walking gaits compared to the baseline ALIP-based MPC approach.