mlsanityData to 13 Oct 2026

Deep Model Predictive Optimization

arXiv 2310.04590 ↗First posted 6 Oct 2023cs.RO cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

A major challenge in robotics is to design robust policies which enable complex and agile behaviors in the real world. On one end of the spectrum, we have model-free reinforcement learning (MFRL), which is incredibly flexible and general but often results in brittle policies. In contrast, model predictive control (MPC) continually re-plans at each time step to remain robust to perturbations and model inaccuracies. However, despite its real-world successes, MPC often under-performs the optimal strategy. This is due to model quality, myopic behavior from short planning horizons, and approximations due to computational constraints. And even with a perfect model and enough compute, MPC can get stuck in bad local optima, depending heavily on the quality of the optimization algorithm. To this end, we propose Deep Model Predictive Optimization (DMPO), which learns the inner-loop of an MPC optimization algorithm directly via experience, specifically tailored to the needs of the control problem. We evaluate DMPO on a real quadrotor agile trajectory tracking task, on which it improves performance over a baseline MPC algorithm for a given computational budget. It can outperform the best MPC algorithm by up to 27% with fewer samples and an end-to-end policy trained with MFRL by 19%. Moreover, because DMPO requires fewer samples, it can also achieve these benefits with 4.3X less memory. When we subject the quadrotor to turbulent wind fields with an attached drag plate, DMPO can adapt zero-shot while still outperforming all baselines. Additional results can be found at https://tinyurl.com/mr2ywmnw.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 79 of its 97 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

79 of 97 references.

References · 79

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2015Trust Region Policy Optimizationcited by 2,513
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,411
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,241
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 754
  9. 2018Learning Dexterous In-Hand Manipulationcited by 645
  10. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 622
  11. 2016Learning to learn by gradient descent by gradient descentcited by 617
  12. 2016Learning to reinforcement learncited by 373
38 more
  1. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 290
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 281
  3. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 270
  4. 2018Residual Reinforcement Learning for Robot Controlcited by 244
  5. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 224
  6. 2016Learning to Optimizecited by 195
  7. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  8. 2018Residual Policy Learningcited by 148
  9. 2017Learned Optimizers that Scale and Generalizecited by 110
  10. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 105
  11. 2022Discovered Policy Optimisationcited by 103
  12. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  13. 2018Evolved Policy Gradientscited by 86
  14. 2018Neural Lander: Stable Drone Landing Control using Learned Dynamicscited by 82
  15. 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
  16. 2017Control of a Quadrotor with Reinforcement Learningcited by 75
  17. 2020Discovering Reinforcement Learning Algorithmscited by 75
  18. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69
  19. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  20. 2020Deep Drone Acrobaticscited by 66
  21. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  22. 2017QMDP-Net: Deep Learning for Planning under Partial Observabilitycited by 51
  23. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  24. 2021Evolving Reinforcement Learning Algorithmscited by 48
  25. 2021Bootstrapped Meta-Learningcited by 45
  26. 2015From Pixels to Torques: Policy Learning with Deep Dynamical Modelscited by 44
  27. 2017Learning to Optimize Neural Netscited by 41
  28. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
  29. 2016Learning Model Predictive Control for iterative tasks. A Data-Driven Control Frameworkcited by 38
  30. 2017Robust Locally-Linear Controllable Embeddingcited by 37
  31. 2019Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotorscited by 36
  32. 2022A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flightcited by 32
  33. 2020The Power of Predictions in Online Controlcited by 31
  34. 2020Training Stronger Baselines for Learning to Optimizecited by 31
  35. 2020Stein Variational Model Predictive Controlcited by 30
  36. 2019An Online Learning Approach to Model Predictive Controlcited by 29
  37. 2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 29
  38. 2019The Differentiable Cross-Entropy Methodcited by 27

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2025Bootstrapped Model Predictive Controlcited by 13
  2. 2024DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learningcited by 6
  3. 2025Model Predictive Adversarial Imitation Learning for Planning from Observationcited by 2
  4. 2024Reinforcement Learning Compensated Model Predictive Control for Off-road Driving on Unknown Deformable Terraincited by 1
  5. 2025Actor-Critic Cooperative Compensation to Model Predictive Control for Off-Road Autonomous Vehicles Under Unknown Dynamicscited by 0
  6. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functionscited by 0
  7. 2026IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproductioncited by 0