# Deep Value Model Predictive Control

arXiv: https://arxiv.org/abs/1910.03358 · HTML: https://mlsanity.com/paper/1910.03358/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 20 of 20)

- [Heuristic-Guided Reinforcement Learning](https://mlsanity.com/paper/2106.02757/) (2021)
- [Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians](https://mlsanity.com/paper/2102.13073/) (2021)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Value-guided action planning with JEPA world models](https://mlsanity.com/paper/2601.00844/) (2025)
- [Optimal Cost Design for Model Predictive Control](https://mlsanity.com/paper/2104.11353/) (2021)
- [Improving Offline RL by Blending Heuristics](https://mlsanity.com/paper/2306.00321/) (2023)
- [Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions](https://mlsanity.com/paper/2307.10524/) (2023)
- [Event-Triggered Model Predictive Control with Deep Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2208.10302/) (2022)
- [A view on learning robust goal-conditioned value functions: Interplay between RL and MPC](https://mlsanity.com/paper/2502.06996/) (2025)
- [CusADi: A GPU Parallelization Framework for Symbolic Expressions and Optimal Control](https://mlsanity.com/paper/2408.09662/) (2024)
- [Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent](https://mlsanity.com/paper/2409.09869/) (2024)
- [Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulation](https://mlsanity.com/paper/2510.14643/) (2025)
- [Value Summation: A Novel Scoring Function for MPC-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2209.08169/) (2022)
- [Local-Global Learning of Interpretable Control Policies: The Interface between MPC and Reinforcement Learning](https://mlsanity.com/paper/2503.13289/) (2025)
- [Incorporating Recurrent Reinforcement Learning into Model Predictive Control for Adaptive Control in Autonomous Driving](https://mlsanity.com/paper/2301.13313/) (2023)
- [Feeding control and water quality monitoring in aquaculture systems: Opportunities and challenges](https://mlsanity.com/paper/2306.09920/) (2023)
- [Bridging RL and MPC for mixed-integer optimal control with application to Formula 1 race strategies](https://mlsanity.com/paper/2604.00826/) (2026)
- [RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments](https://mlsanity.com/paper/2609.39854/) (2026)
- [Order-Consistent Latent Geometry for Reliable Short-Horizon Planning in World Models](https://mlsanity.com/paper/2610.18570/) (2026)

## Abstract (arXiv)

In this paper, we introduce an actor-critic algorithm called Deep Value Model Predictive Control (DMPC), which combines model-based trajectory optimization with value function estimation. The DMPC actor is a Model Predictive Control (MPC) optimizer with an objective function defined in terms of a value function estimated by the critic. We show that our MPC actor is an importance sampler, which minimizes an upper bound of the cross-entropy to the state distribution of the optimal sampling policy. In our experiments with a Ballbot system, we show that our algorithm can work with sparse and binary reward signals to efficiently solve obstacle avoidance and target reaching tasks. Compared to previous work, we show that including the value function in the running cost of the trajectory optimizer speeds up the convergence. We also discuss the necessary strategies to robustify the algorithm in practice.