# Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

arXiv: https://arxiv.org/abs/2606.05687 · HTML: https://mlsanity.com/paper/2606.05687/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 27 of 27)

- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning](https://mlsanity.com/paper/2501.02116/) (2025)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search](https://mlsanity.com/paper/1509.06791/) (2015)
- [Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation](https://mlsanity.com/paper/2409.20514/) (2024)
- [Sim-to-Real Learning for Humanoid Box Loco-Manipulation](https://mlsanity.com/paper/2310.03191/) (2023)
- [Optimization-Based Control for Dynamic Legged Robots](https://mlsanity.com/paper/2211.11644/) (2022)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [mjlab: A Lightweight Framework for GPU-Accelerated Robot Learning](https://mlsanity.com/paper/2601.22074/) (2026)
- [Tailoring Solution Accuracy for Fast Whole-body Model Predictive Control of Legged Robots](https://mlsanity.com/paper/2407.10789/) (2024)
- [Learning Agile Locomotion and Adaptive Behaviors via RL-augmented MPC](https://mlsanity.com/paper/2310.09442/) (2023)
- [Online Non-linear Centroidal MPC for Humanoid Robot Locomotion with Step Adjustment](https://mlsanity.com/paper/2203.04489/) (2022)
- [ReLU-QP: A GPU-Accelerated Quadratic Programming Solver for Model-Predictive Control](https://mlsanity.com/paper/2311.18056/) (2023)
- [Stability of Surface Contacts for Humanoid Robots: Closed-Form Formulae of the Contact Wrench Cone for Rectangular Support Areas](https://mlsanity.com/paper/1501.04719/) (2015)
- [CusADi: A GPU Parallelization Framework for Symbolic Expressions and Optimal Control](https://mlsanity.com/paper/2408.09662/) (2024)
- [Walk the PLANC: Physics-Guided RL for Agile Humanoid Locomotion on Constrained Footholds](https://mlsanity.com/paper/2601.06286/) (2026)
- [RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control](https://mlsanity.com/paper/2407.17683/) (2024)
- [A Differentiable Interior-Point Method in Single Precision](https://mlsanity.com/paper/2605.17913/) (2026)
- [RL-augmented Adaptive Model Predictive Control for Bipedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2509.18466/) (2025)
- [Gait-Net-augmented Implicit Kino-dynamic MPC for Dynamic Variable-frequency Humanoid Locomotion over Discrete Terrains](https://mlsanity.com/paper/2502.02934/) (2025)
- [Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion](https://mlsanity.com/paper/2603.28243/) (2026)
- [OGMP: Oracle Guided Multi-mode Policies for Agile and Versatile Robot Control](https://mlsanity.com/paper/2403.04205/) (2024)
- [πMPC: A Parallel-in-horizon and Construction-free NMPC Solver](https://mlsanity.com/paper/2601.14414/) (2026)

## Built on by (most cited first, 3 of 3)

- [Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation](https://mlsanity.com/paper/2609.14878/) (2026)
- [Opt2VLA: Force-Aware Vision-Language-Action for Contact-Rich Humanoid Whole-Body Manipulation](https://mlsanity.com/paper/2609.23968/) (2026)
- [HULK: Learning Whole-Body Forceful Loco-Manipulation for Humanoids](https://mlsanity.com/paper/2610.08970/) (2026)

## Abstract (arXiv)

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop πnMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.