# Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot

arXiv: https://arxiv.org/abs/2003.03200 · HTML: https://mlsanity.com/paper/2003.03200/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2020.

## Built on by (most cited first, 17 of 17)

- [Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians](https://mlsanity.com/paper/2102.13073/) (2021)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Regularized Newton Method with Global O(1/k2) Convergence](https://mlsanity.com/paper/2112.02089/) (2021)
- [Optimal Cost Design for Model Predictive Control](https://mlsanity.com/paper/2104.11353/) (2021)
- [Safe Model-based Off-policy Reinforcement Learning for Eco-Driving in Connected and Automated Hybrid Electric Vehicles](https://mlsanity.com/paper/2105.11640/) (2021)
- [Event-Triggered Model Predictive Control with Deep Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2208.10302/) (2022)
- [AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control](https://mlsanity.com/paper/2406.03995/) (2024)
- [Regret Analysis of Learning-Based MPC with Partially-Unknown Cost Function](https://mlsanity.com/paper/2108.02307/) (2021)
- [Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent](https://mlsanity.com/paper/2409.09869/) (2024)
- [Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies](https://mlsanity.com/paper/2410.10212/) (2024)
- [Probabilistically safe and efficient model-based reinforcement learning](https://mlsanity.com/paper/2504.00626/) (2025)
- [Learning the References of Online Model Predictive Control for Urban Self-Driving](https://mlsanity.com/paper/2308.15808/) (2023)
- [Out-of-Distribution-Aware Electric Vehicle Charging](https://mlsanity.com/paper/2311.05941/) (2023)
- [A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot](https://mlsanity.com/paper/2409.14867/) (2024)
- [Combining model-predictive control and predictive reinforcement learning for stable quadrupedal robot locomotion](https://mlsanity.com/paper/2307.07752/) (2023)
- [Human-aligned Safe Reinforcement Learning for Highway On-Ramp Merging in Dense Traffic](https://mlsanity.com/paper/2503.02624/) (2025)
- [Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization](https://mlsanity.com/paper/2606.28764/) (2026)

## Abstract (arXiv)

Model Predictive Control (MPC) is a powerful control technique that handles constraints, takes the system's dynamics into account, and optimizes for a given cost function. In practice, however, it often requires an expert to craft and tune this cost function and find trade-offs between different state penalties to satisfy simple high level objectives. In this paper, we use Reinforcement Learning and in particular value learning to approximate the value function given only high level objectives, which can be sparse and binary. Building upon previous works, we present improvements that allowed us to successfully deploy the method on a real world unmanned ground vehicle. Our experiments show that our method can learn the cost function from scratch and without human intervention, while reaching a performance level similar to that of an expert-tuned MPC. We perform a quantitative comparison of these methods with standard MPC approaches both in simulation and on the real robot.