mlsanityData to 7 Oct 2026

Evaluating model-based planning and planner amortization for continuous control

arXiv 2110.03363 ↗First posted 7 Oct 2021cs.RO cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

There is a widespread intuition that model-based control methods should be able to surpass the data efficiency of model-free approaches. In this paper we attempt to evaluate this intuition on various challenging locomotion tasks. We take a hybrid approach, combining model predictive control (MPC) with a learned model and model-free policy learning; the learned policy serves as a proposal for MPC. We find that well-tuned model-free agents are strong baselines even for high DoF control problems but MPC with learned proposals and models (trained on the fly or transferred from related tasks) can significantly improve performance and data efficiency in hard multi-task/multi-goal settings. Finally, we show that it is possible to distil a model-based planner into a policy that amortizes the planning computation without any loss of performance. Videos of agents performing different tasks can be seen at https://sites.google.com/view/mbrl-amortization/home.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

40 of 64 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2018DeepMind Control Suitecited by 935
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
28 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 318
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  3. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  4. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  6. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  7. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  8. 2020Model-Based Offline Planningcited by 126
  9. 2019When to use parametric models in reinforcement learning?cited by 118
  10. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  11. 2019Exploring Model-based Planning with Policy Networkscited by 90
  12. 2021Learning and Planning in Complex Action Spacescited by 76
  13. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  14. 2019Information asymmetry in KL-regularized RLcited by 53
  15. 2020Learning Off-Policy with Online Planningcited by 46
  16. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  17. 2021A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulationcited by 33
  18. 2018Relative Entropy Regularized Policy Iterationcited by 32
  19. 2021Data-Driven MPC for Quadrotorscited by 32
  20. 2019MPC-Net: A First Principles Guided Policy Searchcited by 28
  21. 2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 27
  22. 2021Reverb: A Framework For Experience Replaycited by 27
  23. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  24. 2018Dual Policy Iterationcited by 22
  25. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  26. 2019Feedback MPC for Torque-Controlled Legged Robotscited by 19
  27. 2019Combining Q-Learning and Search with Amortized Value Estimatescited by 19
  28. 2020Local Search for Policy Iteration in Continuous Controlcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 3; 2025: 1; 2026: 3.

15 papers.

Sort:

By area:

  1. 2022Tutorial on amortized optimizationcited by 83
  2. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  3. 2023A Generalist Dynamics Model for Controlcited by 24
  4. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  5. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  6. 2022SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Explorationcited by 6
  7. 2023Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domainscited by 6
  8. 2023Investigating the role of model-based learning in exploration and transfercited by 5
  9. 2024Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learningcited by 2
  10. 2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
  11. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learningcited by 1
  12. 2024Learning Off-policy with Model-based Intrinsic Motivation For Active Online Explorationcited by 1
3 more
  1. 2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imaginationcited by 1
  2. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0
  3. 2026SUN: Agentic Robot Policy Learning with Persistent Task Programscited by 0