mlsanityData to 14 Oct 2026

A Survey on Model-based Reinforcement Learning

arXiv 2206.09328 ↗First posted 19 Jun 2022cs.LGcited by 66

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning (RL) solves sequential decision-making problems via a trial-and-error process interacting with the environment. While RL achieves outstanding success in playing complex video games that allow huge trial-and-error, making errors is always undesired in the real world. To improve the sample efficiency and thus reduce the errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, which builds environment models in which the trial-and-errors can take place without real costs. In this survey, we take a review of MBRL with a focus on the recent progress in deep RL. For non-tabular environments, there is always a generalization error between the learned environment model and the real environment. As such, it is of great importance to analyze the discrepancy between policy training in the environment model and that in the real environment, which in turn guides the algorithm design for better model learning, model usage, and policy training. Besides, we also discuss the recent advances of model-based techniques in other forms of RL, including offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Moreover, we discuss the applicability and advantages of MBRL in real-world tasks. Finally, we end this survey by discussing the promising prospects for the future development of MBRL. We think that MBRL has great potential and advantages in real-world applications that were overlooked, and we hope this survey could attract more research on MBRL.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 117 of its 191 references that are in the map. First posted on arXiv 19 Jun 2022.

Builds on

117 of 191 references.

Closest ancestors · 13

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Causal Structured World Models3 Jun
  2. 2020Mastering Atari with Discrete World Models5 Oct
  3. 2020A Game Theoretic Framework for Model Based Reinforcement Learning16 Apr
  4. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  5. 2019Exploring Model-based Planning with Policy Networks20 Jun
  6. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  7. 2018Recurrent World Models Facilitate Policy Evolution4 Sep
  8. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees10 Jul
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  10. 2018A0C: Alpha Zero in Continuous Action Space24 May
  11. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb
  12. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug
1 more
  1. 2017Value Prediction Network11 Jul

Other references · 104

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2016Generative Adversarial Imitation Learningcited by 1,597
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
38 more
  1. 2017Hindsight Experience Replaycited by 1,136
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  3. 2022A Generalist Agentcited by 808
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  10. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  13. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  14. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  15. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  16. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  17. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  18. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  19. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  20. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  21. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  22. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  23. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  24. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  25. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  26. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  27. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  28. 2016Value Iteration Networkscited by 160
  29. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  30. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  31. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
  32. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 140
  33. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  34. 2019Imitation Learning as f-Divergence Minimizationcited by 123
  35. 2019CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenariocited by 115
  36. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 113
  37. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
  38. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 22; 2024: 14; 2025: 21; 2026: 5.

66 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  2. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  3. 2023A Survey on Causal Reinforcement Learningcited by 45
  4. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  5. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  6. 2023Causal Reinforcement Learning: A Surveycited by 23
  7. 2022SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Renderingcited by 22
  8. 2024World Model-based Perception for Visual Legged Locomotioncited by 18
  9. 2023Reinforcement Learning Approaches for Traffic Signal Control under Missing Datacited by 13
  10. 2023Thinker: Learning to Plan and Actcited by 13
  11. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
  12. 2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 11
38 more
  1. 2025Dream to Drive with Predictive Individual World Modelcited by 11
  2. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  3. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  4. 2025RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operationscited by 9
  5. 2024Learning Control Barrier Functions and their application in Reinforcement Learning: A Surveycited by 8
  6. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  7. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  8. 2023A Survey on Offline Model-Based Reinforcement Learningcited by 7
  9. 2024Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replaycited by 7
  10. 2023Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Drivingcited by 6
  11. 2023Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generationcited by 6
  12. 2023Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systemscited by 5
  13. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 5
  14. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  15. 2024Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videoscited by 4
  16. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learningcited by 4
  17. 2025Tail-Risk-Safe Monte Carlo Tree Search under PAC-Level Guaranteescited by 4
  18. 2025Spatiotemporal Forecasting as Planning: A Model-Based Reinforcement Learning Approach with Generative World Modelscited by 4
  19. 2023VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Modelcited by 3
  20. 2024Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learningcited by 3
  21. 2025Spectral Representation-based Reinforcement Learningcited by 3
  22. 2026Looped World Modelscited by 3
  23. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  24. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarioscited by 2
  25. 2025FLAM: Foundation Model-Based Body Stabilization for Humanoid Locomotion and Manipulationcited by 2
  26. 2025ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rolloutscited by 2
  27. 2025Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Surveycited by 2
  28. 2022Value Summation: A Novel Scoring Function for MPC-based Model-based Reinforcement Learningcited by 1
  29. 2022Off-Policy Deep Reinforcement Learning Algorithms for Handling Various Robotic Manipulator Taskscited by 1
  30. 2023Beware of Instantaneous Dependence in Reinforcement Learningcited by 1
  31. 2023Pretty darn good control: when are approximate solutions better than approximate modelscited by 1
  32. 2023Anytime-Competitive Reinforcement Learning with Policy Priorcited by 1
  33. 2023FOSS: A Self-Learned Doctor for Query Optimizercited by 1
  34. 2023Multi-Agent Probabilistic Ensembles with Trajectory Sampling for Connected Autonomous Vehiclescited by 1
  35. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networkscited by 1
  36. 2024Imitation Game: A Model-based and Imitation Learning Deep Reinforcement Learning Hybridcited by 1
  37. 2024Introduction to Reinforcement Learningcited by 1
  38. 2024Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shippingcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards General-Purpose Model-Free Reinforcement Learning
  2. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  3. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  4. 2026Reinforcement Learning: From Algorithms To Foundation Models
  5. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  6. 2025Scalable Offline Model-Based RL with Action Chunks
  7. 2024Reinforcement Learning: An Overview
  8. 2025First Order Model-Based RL through Decoupled Backpropagation