mlsanityData to 14 Oct 2026

Model-based Reinforcement Learning with Multi-step Plan Value Estimation

arXiv 2209.05530 ↗First posted 12 Sep 2022cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

A promising way to improve the sample efficiency of reinforcement learning is model-based methods, in which many explorations and evaluations can happen in the learned models to save real-world samples. However, when the learned model has a non-negligible model error, sequential steps in the model are hard to be accurately evaluated, limiting the model's utilization. This paper proposes to alleviate this issue by introducing multi-step plans to replace multi-step actions for model-based RL. We employ the multi-step plan value estimation, which evaluates the expected discounted return after executing a sequence of action plans at a given state, and updates the policy by directly computing the multi-step policy gradient via plan value estimation. The new model-based reinforcement learning algorithm MPPVE (Model-based Planning Policy Learning with Multi-step Plan Value Estimation) shows a better utilization of the learned model and achieves a better sample efficiency than state-of-the-art model-based RL approaches.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

20 of 30 references.

References · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  6. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  7. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  8. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  9. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  10. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  11. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 54
  12. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
8 more
  1. 2020Bidirectional Model-based Policy Optimizationcited by 38
  2. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  3. 2019Differentiable Algorithm Networks for Composable Robot Learningcited by 24
  4. 2022Offline Reinforcement Learning with Causal Structured World Modelscited by 19
  5. 2017Path Integral Networks: End-to-End Differentiable Optimal Controlcited by 16
  6. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  7. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  8. 2018Towards a Simple Approach to Multi-step Model-based Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 19
  2. 2024A call for embodied AIcited by 7
  3. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 5
  4. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Modelscited by 4
  5. 2025Controllable Flow Matching for Online Reinforcement Learningcited by 0
  6. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0