mlsanityData to 7 Oct 2026

When to Trust Your Model: Model-Based Policy Optimization

arXiv 1906.08253 ↗First posted 19 Jun 2019cs.LG cs.AI stat.MLcited by 753

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data. In this paper, we study the role of model usage in policy optimization both theoretically and empirically. We first formulate and analyze a model-based reinforcement learning algorithm with a guarantee of monotonic improvement at each step. In practice, this analysis is overly pessimistic and suggests that real off-policy data is always preferable to model-generated on-policy data, but we show that an empirical estimate of model generalization can be incorporated into such analysis to justify model usage. Motivated by this analysis, we then demonstrate that a simple procedure of using short model-generated rollouts branched from real data has the benefits of more complicated model-based algorithms without the usual pitfalls. In particular, this approach surpasses the sample efficiency of prior model-based methods, matches the asymptotic performance of the best model-free algorithms, and scales to horizons that cause other model-based methods to fail entirely.

Key facts. Cited by 753 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 6 of its 45 references that are in the map. First posted on arXiv 19 Jun 2019.

Builds on

6 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees10 Jul
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  3. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb
  4. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug
  5. 2017Value Prediction Network11 Jul

Other references · 1

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 8; 2021: 110; 2022: 115; 2023: 134; 2024: 131; 2025: 121; 2026: 133.

753 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  9. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
  10. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  12. 2023FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Leadcited by 170
38 more
  1. 2022Masked World Models for Visual Controlcited by 169
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  5. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  9. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  10. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  11. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  12. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  13. 2023Synthetic Experience Replaycited by 100
  14. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  15. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  16. 2019Exploring Model-based Planning with Policy Networkscited by 90
  17. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  18. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 87
  19. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  20. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  21. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  22. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  23. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  24. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  25. 2022A Survey on Model-based Reinforcement Learningcited by 65
  26. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  27. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  28. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  29. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  30. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  31. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  32. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  33. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  34. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  35. 2022Skill-based Model-based Reinforcement Learningcited by 50
  36. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  37. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  38. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48

50 most cited of 753 All 753 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Offline Model-Based Reinforcement Learning
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  3. 2022Model Generation with Provable Coverability for Offline Reinforcement Learning
  4. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  5. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  6. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
  7. 2026Offline Policy Optimization with Posterior Sampling
  8. 2024MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning