mlsanityData to 7 Oct 2026

On Effective Scheduling of Model-based Reinforcement Learning

arXiv 2111.08550 ↗First posted 16 Nov 2021cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning has attracted wide attention due to its superior sample efficiency. Despite its impressive success so far, it is still unclear how to appropriately schedule the important hyperparameters to achieve adequate performance, such as the real data ratio for policy optimization in Dyna-style model-based algorithms. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance. Inspired by the analysis, we propose a framework named AutoMBPO to automatically schedule the real data ratio as well as other hyperparameters in training model-based policy optimization (MBPO) algorithm, a representative running case of model-based methods. On several continuous control tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the original one, and the real data ratio schedule found by AutoMBPO shows consistency with our theoretical analysis.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 16 Nov 2021.

Builds on

22 of 45 references.

References · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  8. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  9. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  10. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  11. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  12. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
10 more
  1. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  2. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  3. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  4. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  5. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  6. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  7. 2020Bidirectional Model-based Policy Optimizationcited by 38
  8. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  9. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  10. 2020Bridging Imagination and Reality for Model-Based Deep Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 4; 2025: 1; 2026: 1.

13 papers.

Sort:

By area:

  1. 2022A Survey on Model-based Reinforcement Learningcited by 65
  2. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  3. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  4. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  5. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  6. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  7. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  8. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  9. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  10. 2024DyDiff: Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learningcited by 2
  11. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  12. 2024Heuristic Algorithm-based Action Masking Reinforcement Learning (HAAM-RL) with Ensemble Inference Methodcited by 0
1 more
  1. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0