# MBRL-Lib: A Modular Library for Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2104.10159 · HTML: https://mlsanity.com/paper/2104.10159/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Gaussian Processes for Data-Efficient Learning in Robotics and Control](https://mlsanity.com/paper/1502.02860/) (2015)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning](https://mlsanity.com/paper/2102.13651/) (2021)
- [On the role of planning in model-based deep reinforcement learning](https://mlsanity.com/paper/2011.04021/) (2020)
- [Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic](https://mlsanity.com/paper/1901.02705/) (2019)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018)
- [The Differentiable Cross-Entropy Method](https://mlsanity.com/paper/1909.12830/) (2019)
- [Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models](https://mlsanity.com/paper/1910.04142/) (2019)
- [Learning model-based planning from scratch](https://mlsanity.com/paper/1707.06170/) (2017)
- [Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow](https://mlsanity.com/paper/2103.14407/) (2021)
- [Baconian: A Unified Open-source Framework for Model-Based Reinforcement Learning](https://mlsanity.com/paper/1904.10762/) (2019)

## Built on by (most cited first, 30 of 42)

- [Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation](https://mlsanity.com/paper/2202.06558/) (2022)
- [Revisiting Design Choices in Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2110.04135/) (2021)
- [Value Gradient weighted Model-Based Reinforcement Learning](https://mlsanity.com/paper/2204.01464/) (2022)
- [Hybrid Inverse Reinforcement Learning](https://mlsanity.com/paper/2402.08848/) (2024)
- [Training Efficient Controllers via Analytic Policy Gradient](https://mlsanity.com/paper/2209.13052/) (2022)
- [Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic](https://mlsanity.com/paper/2306.02865/) (2023)
- [stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation](https://mlsanity.com/paper/2605.21800/) (2026)
- [Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption](https://mlsanity.com/paper/2405.19014/) (2024)
- [An Experimental Design Perspective on Model-Based Reinforcement Learning](https://mlsanity.com/paper/2112.05244/) (2021)
- [Model-Based Uncertainty in Value Functions](https://mlsanity.com/paper/2302.12526/) (2023)
- [When to Update Your Model: Constrained Model-based Reinforcement Learning](https://mlsanity.com/paper/2210.08349/) (2022)
- [The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms](https://mlsanity.com/paper/2303.00694/) (2023)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.06335/) (2023)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [Pearl: A Production-ready Reinforcement Learning Agent](https://mlsanity.com/paper/2312.03814/) (2023)
- [Exploration via Planning for Information about the Optimal Trajectory](https://mlsanity.com/paper/2210.04642/) (2022)
- [Continual Reinforcement Learning by Planning with Online World Models](https://mlsanity.com/paper/2507.09177/) (2025)
- [Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms](https://mlsanity.com/paper/2310.19927/) (2023)
- [Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement](https://mlsanity.com/paper/2303.11373/) (2023)
- [Value-Distributional Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.06590/) (2023)
- [Continuous Monte Carlo Graph Search](https://mlsanity.com/paper/2210.01426/) (2022)
- [Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting](https://mlsanity.com/paper/2303.10144/) (2023)
- [Data-efficient Deep Reinforcement Learning for Vehicle Trajectory Control](https://mlsanity.com/paper/2311.18393/) (2023)
- [Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning](https://mlsanity.com/paper/2111.04972/) (2021)
- [Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation](https://mlsanity.com/paper/2301.11374/) (2023)
- [Goal-conditioned Offline Planning from Curious Exploration](https://mlsanity.com/paper/2311.16996/) (2023)
- [Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization](https://mlsanity.com/paper/2312.04386/) (2023)
- [Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling](https://mlsanity.com/paper/2410.04988/) (2024)

## Abstract (arXiv)

Model-based reinforcement learning is a compelling framework for data-efficient learning of agents that interact with the world. This family of algorithms has many subcomponents that need to be carefully selected and tuned. As a result the entry-bar for researchers to approach the field and to deploy it in real-world tasks can be daunting. In this paper, we present MBRL-Lib -- a machine learning library for model-based reinforcement learning in continuous state-action spaces based on PyTorch. MBRL-Lib is designed as a platform for both researchers, to easily develop, debug and compare new algorithms, and non-expert user, to lower the entry-bar of deploying state-of-the-art algorithms. MBRL-Lib is open-source at https://github.com/facebookresearch/mbrl-lib.