# A Survey on Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2206.09328 · HTML: https://mlsanity.com/paper/2206.09328/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 117 of its 191 references that are in the map.
- First posted on arXiv 19 Jun 2022.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 30 of 117)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)

## Built on by (most cited first, 30 of 66)

- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering](https://mlsanity.com/paper/2210.15185/) (2022)
- [World Model-based Perception for Visual Legged Locomotion](https://mlsanity.com/paper/2409.16784/) (2024)
- [Reinforcement Learning Approaches for Traffic Signal Control under Missing Data](https://mlsanity.com/paper/2304.10722/) (2023)
- [Thinker: Learning to Plan and Act](https://mlsanity.com/paper/2307.14993/) (2023)
- [Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.05422/) (2023)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [Dream to Drive with Predictive Individual World Model](https://mlsanity.com/paper/2501.16733/) (2025)
- [AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors](https://mlsanity.com/paper/2403.09976/) (2024)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations](https://mlsanity.com/paper/2503.23101/) (2025)
- [Learning Control Barrier Functions and their application in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2404.16879/) (2024)
- [G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration](https://mlsanity.com/paper/2506.09272/) (2025)
- [FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making](https://mlsanity.com/paper/2507.12496/) (2025)
- [A Survey on Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2305.03360/) (2023)
- [Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay](https://mlsanity.com/paper/2411.10809/) (2024)
- [Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving](https://mlsanity.com/paper/2302.03939/) (2023)
- [Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generation](https://mlsanity.com/paper/2305.05116/) (2023)
- [Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems](https://mlsanity.com/paper/2305.04832/) (2023)
- [Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations](https://mlsanity.com/paper/2512.21586/) (2025)
- [DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning](https://mlsanity.com/paper/2309.08925/) (2023)
- [Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos](https://mlsanity.com/paper/2412.10778/) (2024)
- [Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning](https://mlsanity.com/paper/2412.14312/) (2024)
- [Tail-Risk-Safe Monte Carlo Tree Search under PAC-Level Guarantees](https://mlsanity.com/paper/2508.05441/) (2025)
- [Spatiotemporal Forecasting as Planning: A Model-Based Reinforcement Learning Approach with Generative World Models](https://mlsanity.com/paper/2510.04020/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) solves sequential decision-making problems via a trial-and-error process interacting with the environment. While RL achieves outstanding success in playing complex video games that allow huge trial-and-error, making errors is always undesired in the real world. To improve the sample efficiency and thus reduce the errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, which builds environment models in which the trial-and-errors can take place without real costs. In this survey, we take a review of MBRL with a focus on the recent progress in deep RL. For non-tabular environments, there is always a generalization error between the learned environment model and the real environment. As such, it is of great importance to analyze the discrepancy between policy training in the environment model and that in the real environment, which in turn guides the algorithm design for better model learning, model usage, and policy training. Besides, we also discuss the recent advances of model-based techniques in other forms of RL, including offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Moreover, we discuss the applicability and advantages of MBRL in real-world tasks. Finally, we end this survey by discussing the promising prospects for the future development of MBRL. We think that MBRL has great potential and advantages in real-world applications that were overlooked, and we hope this survey could attract more research on MBRL.