# Deep Model Predictive Optimization

arXiv: https://arxiv.org/abs/2310.04590 · HTML: https://mlsanity.com/paper/2310.04590/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 97 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 30 of 79)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Understanding and correcting pathologies in the training of learned optimizers](https://mlsanity.com/paper/1810.10180/) (2018)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Evolved Policy Gradients](https://mlsanity.com/paper/1802.04821/) (2018)
- [Neural Lander: Stable Drone Landing Control using Learned Dynamics](https://mlsanity.com/paper/1811.08027/) (2018)
- [Neural-Fly Enables Rapid Learning for Agile Flight in Strong Winds](https://mlsanity.com/paper/2205.06908/) (2022)
- [Understanding Domain Randomization for Sim-to-real Transfer](https://mlsanity.com/paper/2110.03239/) (2021)
- [Discovering Reinforcement Learning Algorithms](https://mlsanity.com/paper/2007.08794/) (2020)

## Built on by (most cited first, 7 of 7)

- [Bootstrapped Model Predictive Control](https://mlsanity.com/paper/2503.18871/) (2025)
- [DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning](https://mlsanity.com/paper/2402.05421/) (2024)
- [Model Predictive Adversarial Imitation Learning for Planning from Observation](https://mlsanity.com/paper/2507.21533/) (2025)
- [Reinforcement Learning Compensated Model Predictive Control for Off-road Driving on Unknown Deformable Terrain](https://mlsanity.com/paper/2408.09253/) (2024)
- [Actor-Critic Cooperative Compensation to Model Predictive Control for Off-Road Autonomous Vehicles Under Unknown Dynamics](https://mlsanity.com/paper/2503.00577/) (2025)
- [Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions](https://mlsanity.com/paper/2510.06649/) (2025)
- [IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction](https://mlsanity.com/paper/2603.29315/) (2026)

## Abstract (arXiv)

A major challenge in robotics is to design robust policies which enable complex and agile behaviors in the real world. On one end of the spectrum, we have model-free reinforcement learning (MFRL), which is incredibly flexible and general but often results in brittle policies. In contrast, model predictive control (MPC) continually re-plans at each time step to remain robust to perturbations and model inaccuracies. However, despite its real-world successes, MPC often under-performs the optimal strategy. This is due to model quality, myopic behavior from short planning horizons, and approximations due to computational constraints. And even with a perfect model and enough compute, MPC can get stuck in bad local optima, depending heavily on the quality of the optimization algorithm. To this end, we propose Deep Model Predictive Optimization (DMPO), which learns the inner-loop of an MPC optimization algorithm directly via experience, specifically tailored to the needs of the control problem. We evaluate DMPO on a real quadrotor agile trajectory tracking task, on which it improves performance over a baseline MPC algorithm for a given computational budget. It can outperform the best MPC algorithm by up to 27% with fewer samples and an end-to-end policy trained with MFRL by 19%. Moreover, because DMPO requires fewer samples, it can also achieve these benefits with 4.3X less memory. When we subject the quadrotor to turbulent wind fields with an attached drag plate, DMPO can adapt zero-shot while still outperforming all baselines. Additional results can be found at https://tinyurl.com/mr2ywmnw.