# Model-based Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2006.16712 · HTML: https://mlsanity.com/paper/2006.16712/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 30 of 39)

- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control](https://mlsanity.com/paper/2108.00128/) (2021)
- [SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning](https://mlsanity.com/paper/2111.14693/) (2021)
- [Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy](https://mlsanity.com/paper/2406.16087/) (2024)
- [Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2010.12142/) (2020)
- [Decision-Making Under Uncertainty: Beyond Probabilities](https://mlsanity.com/paper/2303.05848/) (2023)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control](https://mlsanity.com/paper/2408.17380/) (2024)
- [PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution](https://mlsanity.com/paper/2607.16636/) (2026)
- [Learning from Data Streams: An Overview and Update](https://mlsanity.com/paper/2212.14720/) (2022)
- [A Definition and Roadmap for World Models](https://mlsanity.com/paper/2607.06401/) (2026)
- [MABL: Bi-Level Latent-Variable World Model for Sample-Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.06011/) (2023)
- [Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review](https://mlsanity.com/paper/2301.10067/) (2023)
- [Differentiable Information Enhanced Model-Based Reinforcement Learning](https://mlsanity.com/paper/2503.01178/) (2025)
- [GEM: Group Enhanced Model for Learning Dynamical Control Systems](https://mlsanity.com/paper/2104.02844/) (2021)
- [A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making](https://mlsanity.com/paper/2304.10590/) (2023)
- [Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation](https://mlsanity.com/paper/2403.07261/) (2024)
- [Reinforcement Learning and Tree Search Methods for the Unit Commitment Problem](https://mlsanity.com/paper/2212.06001/) (2022)
- [Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewards](https://mlsanity.com/paper/2403.19024/) (2024)
- [Explainable Fuzzy Neural Network with Multi-Fidelity Reinforcement Learning for Micro-Architecture Design Space Exploration](https://mlsanity.com/paper/2412.10754/) (2024)
- [Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Models](https://mlsanity.com/paper/2502.10012/) (2025)
- [Best-Effort Policies for Robust Markov Decision Processes](https://mlsanity.com/paper/2508.07790/) (2025)
- [A Unified Framework for Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.20542/) (2025)
- [Joint Learning of Hierarchical Neural Options and Abstract World Model](https://mlsanity.com/paper/2602.02799/) (2026)
- [Sensor Control for Information Gain in Dynamic, Sparse and Partially Observed Environments](https://mlsanity.com/paper/2211.01527/) (2022)
- [Reinforcement Learning in the Wild with Maximum Likelihood-based Model Transfer](https://mlsanity.com/paper/2302.09273/) (2023)
- [Towards a Unified Framework for Sequential Decision Making](https://mlsanity.com/paper/2310.02167/) (2023)
- [Virtual Action Actor-Critic Framework for Exploration (Student Abstract)](https://mlsanity.com/paper/2311.02916/) (2023)
- [Efficient Parallel Reinforcement Learning Framework using the Reactor Model](https://mlsanity.com/paper/2312.04704/) (2023)

## Abstract (arXiv)

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is a important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This paper presents a survey of the integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer learning. Altogether, the survey presents a broad conceptual overview of the combination of planning and learning for MDP optimization.