# Asynchronous Methods for Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/1910.12453 · HTML: https://mlsanity.com/paper/1910.12453/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models](https://mlsanity.com/paper/2301.04741/) (2023)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challenges](https://mlsanity.com/paper/2508.09561/) (2025)
- [Federated Ensemble Model-based Reinforcement Learning in Edge Computing](https://mlsanity.com/paper/2109.05549/) (2021)
- [Self-reconfiguration Strategies for Space-distributed Spacecraft](https://mlsanity.com/paper/2411.17137/) (2024)

## Abstract (arXiv)

Significant progress has been made in the area of model-based reinforcement learning. State-of-the-art algorithms are now able to match the asymptotic performance of model-free methods while being significantly more data efficient. However, this success has come at a price: state-of-the-art model-based methods require significant computation interleaved with data collection, resulting in run times that take days, even if the amount of agent interaction might be just hours or even minutes. When considering the goal of learning in real-time on real robots, this means these state-of-the-art model-based algorithms still remain impractical. In this work, we propose an asynchronous framework for model-based reinforcement learning methods that brings down the run time of these algorithms to be just the data collection time. We evaluate our asynchronous framework on a range of standard MuJoCo benchmarks. We also evaluate our asynchronous framework on three real-world robotic manipulation tasks. We show how asynchronous learning not only speeds up learning w.r.t wall-clock time through parallelization, but also further reduces the sample complexity of model-based approaches by means of improving the exploration and by means of effectively avoiding the policy overfitting to the deficiencies of learned dynamics models.