mlsanityData to 7 Oct 2026

Exploring Model-based Planning with Policy Networks

arXiv 1906.08649 ↗First posted 20 Jun 2019cs.LG cs.AI cs.ROcited by 90

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in terms of both sample efficiency and asymptotic performance. Despite their initial successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments, being about 3x more sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released in https://github.com/WilsonWangTHU/POPLIN.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 9 of its 44 references that are in the map. First posted on arXiv 20 Jun 2019.

Builds on

9 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  2. 2018Recurrent World Models Facilitate Policy Evolution4 Sep
  3. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees10 Jul
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  5. 2018World Models27 Mar
  6. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug

Other references · 2

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 4; 2021: 23; 2022: 23; 2023: 11; 2024: 12; 2025: 10; 2026: 6.

90 papers.

Sort:

By area:

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2020Mastering Atari with Discrete World Modelscited by 990
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  8. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  9. 2022Tutorial on amortized optimizationcited by 83
  10. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  11. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  12. 2024Simple Hierarchical Planning with Diffusioncited by 71
38 more
  1. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  2. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  3. 2022A Survey on Model-based Reinforcement Learningcited by 65
  4. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  5. 2022Skill-based Model-based Reinforcement Learningcited by 50
  6. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  7. 2022Deconstructing the Inductive Biases of Hamiltonian Neural Networkscited by 31
  8. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  9. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  10. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  11. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  12. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  13. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  14. 2021Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrianscited by 20
  15. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 20
  16. 2025Learning Transformer-based World Models with Contrastive Predictive Codingcited by 20
  17. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  18. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  19. 2021Model-Invariant State Abstractions for Model-Based Reinforcement Learningcited by 18
  20. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  21. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  22. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  23. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  24. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  25. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  26. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  27. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  28. 2021MapGo: Model-Assisted Policy Optimization for Goal-Oriented Taskscited by 13
  29. 2024MuDreamer: Learning Predictive World Models without Reconstructioncited by 13
  30. 2021Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?cited by 11
  31. 2024Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learningcited by 11
  32. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulatorscited by 10
  33. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  34. 2022Learning Sampling Distributions for Model Predictive Controlcited by 10
  35. 2022Learning to Optimize in Model Predictive Controlcited by 10
  36. 2024Efficient Multi-agent Reinforcement Learning by Planningcited by 9
  37. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  38. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024PWM: Policy Learning with Multi-Task World Models
  2. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  3. 2025Bootstrapped Model Predictive Control
  4. 2020Ready Policy One: World Building Through Active Learning
  5. 2025Flexible Locomotion Learning with Diffusion Model Predictive Control
  6. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics
  7. 2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
  8. 2024Diffusion-based learning of contact plans for agile locomotion ↗