# RLlib: Abstractions for Distributed Reinforcement Learning

arXiv: https://arxiv.org/abs/1712.09381 · HTML: https://mlsanity.com/paper/1712.09381/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Dec 2017.

## Built on by (most cited first, 30 of 54)

- [RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark](https://mlsanity.com/paper/2306.17100/) (2023)
- [DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents](https://mlsanity.com/paper/2410.14803/) (2024)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [The Neural MMO Platform for Massively Multiagent Research](https://mlsanity.com/paper/2110.07594/) (2021)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [gym-DSSAT: a crop model turned into a Reinforcement Learning environment](https://mlsanity.com/paper/2207.03270/) (2022)
- [A Review of Reinforcement Learning in Financial Applications](https://mlsanity.com/paper/2411.12746/) (2024)
- [Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximization](https://mlsanity.com/paper/2110.04686/) (2021)
- [Target Entropy Annealing for Discrete Soft Actor-Critic](https://mlsanity.com/paper/2112.02852/) (2021)
- [Consistent Dropout for Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/2202.11818/) (2022)
- [SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores](https://mlsanity.com/paper/2306.16688/) (2023)
- [LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations](https://mlsanity.com/paper/1808.07903/) (2018)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [POGEMA: Partially Observable Grid Environment for Multiple Agents](https://mlsanity.com/paper/2206.10944/) (2022)
- [Goal-conditioned reinforcement learning for ultrasound navigation guidance](https://mlsanity.com/paper/2405.01409/) (2024)
- [Machine Learning aided Crop Yield Optimization](https://mlsanity.com/paper/2111.00963/) (2021)
- [Centrally Coordinated Multi-Agent Reinforcement Learning for Power Grid Topology Control](https://mlsanity.com/paper/2502.08681/) (2025)
- [MarsExplorer: Exploration of Unknown Terrains via Deep Reinforcement Learning and Procedurally Generated Environments](https://mlsanity.com/paper/2107.09996/) (2021)
- [Self-Improving Safety Performance of Reinforcement Learning Based Driving with Black-Box Verification Algorithms](https://mlsanity.com/paper/2210.16575/) (2022)
- [Exposure-Based Multi-Agent Inspection of a Tumbling Target Using Deep Reinforcement Learning](https://mlsanity.com/paper/2302.14188/) (2023)
- [Deep Multi-Agent Reinforcement Learning for Decentralized Active Hypothesis Testing](https://mlsanity.com/paper/2309.08477/) (2023)
- [Improved Cooperation by Exploiting a Common Signal](https://mlsanity.com/paper/2102.02304/) (2021)
- [AI-driven Prices for Externalities and Sustainability in Production Markets](https://mlsanity.com/paper/2106.06060/) (2021)
- [Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operations](https://mlsanity.com/paper/2106.11345/) (2021)
- [Creating Valid Adversarial Examples of Malware](https://mlsanity.com/paper/2306.13587/) (2023)
- [Modeling Boundedly Rational Agents with Latent Inference Budgets](https://mlsanity.com/paper/2312.04030/) (2023)
- [Explanation-Aware Experience Replay in Rule-Dense Environments](https://mlsanity.com/paper/2109.14711/) (2021)
- [Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations](https://mlsanity.com/paper/2110.01101/) (2021)
- [Decentralized Cooperative Lane Changing at Freeway Weaving Areas Using Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2110.08124/) (2021)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation. We argue for distributing RL components in a composable way by adapting algorithms for top-down hierarchical control, thereby encapsulating parallelism and resource requirements within short-running compute tasks. We demonstrate the benefits of this principle through RLlib: a library that provides scalable software primitives for RL. These primitives enable a broad range of algorithms to be implemented with high performance, scalability, and substantial code reuse. RLlib is available at https://rllib.io/.