# Benchmarking Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.02057 · HTML: https://mlsanity.com/paper/1907.02057/

## Key facts

- Cited by 162 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2019.

## Built on by (most cited first, 30 of 162)

- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [What Matters for Adversarial Imitation Learning?](https://mlsanity.com/paper/2106.00672/) (2021)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning](https://mlsanity.com/paper/2102.13651/) (2021)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2202.10324/) (2022)
- [Making Linear MDPs Practical via Contrastive Representation Learning](https://mlsanity.com/paper/2207.07150/) (2022)
- [safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://mlsanity.com/paper/2109.06325/) (2021)
- [Investigating Compounding Prediction Errors in Learned Dynamics Models](https://mlsanity.com/paper/2203.09637/) (2022)
- [MobILE: Model-Based Imitation Learning From Observation Alone](https://mlsanity.com/paper/2102.10769/) (2021)
- [MBRL-Lib: A Modular Library for Model-based Reinforcement Learning](https://mlsanity.com/paper/2104.10159/) (2021)
- [Physical Design using Differentiable Learned Simulators](https://mlsanity.com/paper/2202.00728/) (2022)
- [Spectral Decomposition Representation for Reinforcement Learning](https://mlsanity.com/paper/2208.09515/) (2022)
- [World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges](https://mlsanity.com/paper/2301.05832/) (2023)
- [Deep reinforcement learning in medical imaging: A literature review](https://mlsanity.com/paper/2103.05115/) (2021)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization](https://mlsanity.com/paper/2104.13877/) (2021)
- [Procedural Generalization by Planning with Self-Supervised World Models](https://mlsanity.com/paper/2111.01587/) (2021)
- [Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.03570/) (2024)
- [Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm](https://mlsanity.com/paper/2210.07573/) (2022)
- [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023)
- [CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning](https://mlsanity.com/paper/2310.12972/) (2023)
- [Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective](https://mlsanity.com/paper/2209.08466/) (2022)
- [RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization](https://mlsanity.com/paper/2307.10224/) (2023)
- [Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2011.01734/) (2020)
- [A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2106.02097/) (2021)

## Abstract (arXiv)

Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL. However, research in model-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate lines of research, which are sometimes closed-sourced or not reproducible. Accordingly, it is an open question how these various existing MBRL algorithms perform relative to each other. To facilitate research in MBRL, in this paper we gather a wide collection of MBRL algorithms and propose over 18 benchmarking environments specially designed for MBRL. We benchmark these algorithms with unified problem settings, including noisy environments. Beyond cataloguing performance, we explore and unify the underlying algorithmic differences across MBRL algorithms. We characterize three key research challenges for future MBRL research: the dynamics bottleneck, the planning horizon dilemma, and the early-termination dilemma. Finally, to maximally facilitate future research on MBRL, we open-source our benchmark in http://www.cs.toronto.edu/~tingwuwang/mbrl.html.