# Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control

arXiv: https://arxiv.org/abs/2510.12717 · HTML: https://mlsanity.com/paper/2510.12717/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer](https://mlsanity.com/paper/2404.05695/) (2024)
- [Feedback Control For Cassie With Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05580/) (2018)
- [Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation](https://mlsanity.com/paper/2409.20514/) (2024)
- [Viser: Imperative, Web-based 3D Visualization in Python](https://mlsanity.com/paper/2507.22885/) (2025)
- [Learning Dynamics Models for Model Predictive Agents](https://mlsanity.com/paper/2109.14311/) (2021)
- [Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model](https://mlsanity.com/paper/2010.11234/) (2020)
- [Benchmarking Potential Based Rewards for Learning Humanoid Locomotion](https://mlsanity.com/paper/2307.10142/) (2023)
- [Representation-Free Model Predictive Control for Dynamic Motions in Quadrupeds](https://mlsanity.com/paper/2012.10002/) (2020)
- [Tailoring Solution Accuracy for Fast Whole-body Model Predictive Control of Legged Robots](https://mlsanity.com/paper/2407.10789/) (2024)
- [Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation](https://mlsanity.com/paper/2410.03076/) (2024)
- [Learning Agile Locomotion and Adaptive Behaviors via RL-augmented MPC](https://mlsanity.com/paper/2310.09442/) (2023)
- [Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion](https://mlsanity.com/paper/2311.02304/) (2023)
- [Integrating Model-Based Footstep Planning with Model-Free Reinforcement Learning for Dynamic Legged Locomotion](https://mlsanity.com/paper/2408.02662/) (2024)
- [RAMBO: RL-Augmented Model-Based Whole-Body Control for Loco-Manipulation](https://mlsanity.com/paper/2504.06662/) (2025)
- [Humanoid Self-Collision Avoidance Using Whole-Body Control with Control Barrier Functions](https://mlsanity.com/paper/2207.00692/) (2022)
- [RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control](https://mlsanity.com/paper/2407.17683/) (2024)
- [RL-augmented Adaptive Model Predictive Control for Bipedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2509.18466/) (2025)
- [Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.04140/) (2025)

## Built on by (most cited first, 3 of 3)

- [MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins](https://mlsanity.com/paper/2606.26392/) (2026)
- [Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation](https://mlsanity.com/paper/2609.21307/) (2026)
- [Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning](https://mlsanity.com/paper/2610.06019/) (2026)

## Abstract (arXiv)

Model Predictive Control (MPC) provides interpretable, tunable locomotion controllers grounded in physical models, but its robustness depends on frequent replanning and is limited by model mismatch and real-time computational constraints. Reinforcement Learning (RL), by contrast, can produce highly robust behaviors through stochastic training but often lacks interpretability, suffers from out-of-distribution failures, and requires intensive reward engineering. This work presents a GPU-parallelized residual architecture that tightly integrates MPC and RL by blending their outputs at the torque-control level. We develop a kinodynamic whole-body MPC formulation evaluated across thousands of agents in parallel at 100 Hz for RL training. The residual policy learns to make targeted corrections to the MPC outputs, combining the interpretability and constraint handling of model-based control with the adaptability of RL. The model-based control prior acts as a strong bias, initializing and guiding the policy towards desirable behavior with a simple set of rewards. Compared to standalone MPC or end-to-end RL, our approach achieves higher sample efficiency, converges to greater asymptotic rewards, expands the range of trackable velocity commands, and enables zero-shot adaptation to unseen gaits and uneven terrain.