# TorchBeast: A PyTorch Platform for Distributed RL

arXiv: https://arxiv.org/abs/1910.03552 · HTML: https://mlsanity.com/paper/1910.03552/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 30 of 41)

- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06135/) (2021)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [A Modern Self-Referential Weight Matrix That Learns to Modify Itself](https://mlsanity.com/paper/2202.05780/) (2022)
- [CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents](https://mlsanity.com/paper/2110.10067/) (2021)
- [Interesting Object, Curious Agent: Learning Task-Agnostic Exploration](https://mlsanity.com/paper/2111.13119/) (2021)
- [Insights From the NeurIPS 2021 NetHack Challenge](https://mlsanity.com/paper/2203.11889/) (2022)
- [Large Batch Simulation for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.07013/) (2021)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [AutoShard: Automated Embedding Table Sharding for Recommender Systems](https://mlsanity.com/paper/2208.06399/) (2022)
- [Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds](https://mlsanity.com/paper/2210.13417/) (2022)
- [Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning](https://mlsanity.com/paper/2402.03046/) (2024)
- [SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark](https://mlsanity.com/paper/2110.10661/) (2021)
- [SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores](https://mlsanity.com/paper/2306.16688/) (2023)
- [minimax: Efficient Baselines for Autocurricula in JAX](https://mlsanity.com/paper/2311.12716/) (2023)
- [Improving Policy Learning via Language Dynamics Distillation](https://mlsanity.com/paper/2210.00066/) (2022)
- [Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning](https://mlsanity.com/paper/2311.03736/) (2023)
- [Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness](https://mlsanity.com/paper/2105.09992/) (2021)
- [Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning](https://mlsanity.com/paper/2207.11584/) (2022)
- [Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform](https://mlsanity.com/paper/2310.00036/) (2023)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Games](https://mlsanity.com/paper/2209.10055/) (2022)
- [Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox](https://mlsanity.com/paper/2212.00253/) (2022)
- [Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision](https://mlsanity.com/paper/2102.13565/) (2021)
- [SiT: Symmetry-Invariant Transformers for Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2406.15025/) (2024)
- [Grid-to-Graph: Flexible Spatial Relational Inductive Biases for Reinforcement Learning](https://mlsanity.com/paper/2102.04220/) (2021)

## Abstract (arXiv)

TorchBeast is a platform for reinforcement learning (RL) research in PyTorch. It implements a version of the popular IMPALA algorithm for fast, asynchronous, parallel training of RL agents. Additionally, TorchBeast has simplicity as an explicit design goal: We provide both a pure-Python implementation ("MonoBeast") as well as a multi-machine high-performance version ("PolyBeast"). In the latter, parts of the implementation are written in C++, but all parts pertaining to machine learning are kept in simple Python using PyTorch, with the environments provided using the OpenAI Gym interface. This enables researchers to conduct scalable RL research using TorchBeast without any programming knowledge beyond Python and PyTorch. In this paper, we describe the TorchBeast design principles and implementation and demonstrate that it performs on-par with IMPALA on Atari. TorchBeast is released as an open-source package under the Apache 2.0 license and is available at urlhttps://github.com/facebookresearch/torchbeast.