# Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

arXiv: https://arxiv.org/abs/2603.28243 · HTML: https://mlsanity.com/paper/2603.28243/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 9 of 9)

- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning](https://mlsanity.com/paper/2501.02116/) (2025)
- [A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation](https://mlsanity.com/paper/2103.00946/) (2021)
- [Learning Agile Bipedal Motions on a Quadrupedal Robot](https://mlsanity.com/paper/2311.05818/) (2023)
- [MPC-based Reinforcement Learning for a Simplified Freight Mission of Autonomous Surface Vehicles](https://mlsanity.com/paper/2106.08634/) (2021)
- [Planar Bipedal Locomotion with Nonlinear Model Predictive Control: Online Gait Generation using Whole-Body Dynamics](https://mlsanity.com/paper/2203.07429/) (2022)
- [Beyond Inverted Pendulums: Task-optimal Simple Models of Legged Locomotion](https://mlsanity.com/paper/2301.02075/) (2023)

## Built on by (most cited first, 2 of 2)

- [Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation](https://mlsanity.com/paper/2606.05687/) (2026)
- [Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC](https://mlsanity.com/paper/2609.01061/) (2026)

## Abstract (arXiv)

In this paper, we propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning (RL) framework. A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data. Specifically, the MPC cost-to-go is evaluated along recorded state-action trajectories, and the parameters are updated to minimize the discrepancy between MPC-predicted values and measured returns. This formulation enables efficient gradient-based learning while avoiding the computational burden of repeatedly solving the MPC problem during training. The proposed method is validated in simulation using a commercial humanoid platform. Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances compared with manually tuned baselines.