mlsanityData to 7 Oct 2026

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

arXiv 1807.03858 ↗First posted 10 Jul 2018cs.LG cs.AI stat.MLcited by 138

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires textitno explicit uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves state-of-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 6 of its 75 references that are in the map. First posted on arXiv 10 Jul 2018.

Builds on

6 of 75 references.

Closest ancestors · 5

Same direction, newest first

  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  2. 2018World Models27 Mar
  3. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb
  4. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug
  5. 2017Value Prediction Network11 Jul

Other references · 1

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 3; 2020: 5; 2021: 35; 2022: 25; 2023: 25; 2024: 27; 2025: 10; 2026: 8.

138 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  4. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  5. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  6. 2019Exploring Model-based Planning with Policy Networkscited by 90
  7. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  8. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  9. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  10. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  11. 2022A Survey on Model-based Reinforcement Learningcited by 65
  12. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
38 more
  1. 2023Behavior Proximal Policy Optimizationcited by 43
  2. 2021On the Sample Complexity of Stability Constrained Imitation Learningcited by 38
  3. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  4. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  5. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  6. 2021Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvaturecited by 32
  7. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  8. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 30
  9. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  10. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  11. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  12. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Planscited by 27
  13. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  14. 2021Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rolloutscited by 24
  15. 2021A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learningcited by 24
  16. 2023Learning Energy Decompositions for Partial Inference of GFlowNetscited by 24
  17. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  18. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  19. 2021Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Controlcited by 22
  20. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guaranteescited by 22
  21. 2021Model-Based Opponent Modelingcited by 21
  22. 2023Deep Learning in Deterministic Computational Mechanicscited by 21
  23. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 21
  24. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  25. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  26. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 18
  27. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  28. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  29. 2024World Model-based Perception for Visual Legged Locomotioncited by 17
  30. 2021Minimax Model Learningcited by 16
  31. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 16
  32. 2023Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learningcited by 16
  33. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  34. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  35. 2022Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computationcited by 14
  36. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  37. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  38. 2022Latent Variable Representation for Reinforcement Learningcited by 14

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds
  2. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
  3. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  4. 2026Maximum Likelihood Reinforcement Learning
  5. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  6. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  7. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning
  8. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic