# RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem

arXiv: https://arxiv.org/abs/2011.12719 · HTML: https://mlsanity.com/paper/2011.12719/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Laminar: A Scalable Asynchronous RL Post-Training Framework](https://mlsanity.com/paper/2510.12633/) (2025)
- [MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster](https://mlsanity.com/paper/2507.19017/) (2025)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems](https://mlsanity.com/paper/2507.14850/) (2025)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [Gym-V: A Unified Vision Environment System for Agentic Vision Research](https://mlsanity.com/paper/2603.15432/) (2026)
- [The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning](https://mlsanity.com/paper/2607.01415/) (2026)
- [Efficient Parallel Reinforcement Learning Framework using the Reactor Model](https://mlsanity.com/paper/2312.04704/) (2023)
- [How to craft a deep reinforcement learning policy for wind farm flow control](https://mlsanity.com/paper/2506.06204/) (2025)
- [Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts](https://mlsanity.com/paper/2507.07348/) (2025)
- [MuFlex: A Scalable, Physics-based Platform for Multi-Building Flexibility Analysis and Coordination](https://mlsanity.com/paper/2508.13532/) (2025)
- [DAPO: Design Structure-Aware Pass Ordering in High-Level Synthesis with Graph Contrastive and Reinforcement Learning](https://mlsanity.com/paper/2512.11342/) (2025)
- [Asynchronous MultiAgent Reinforcement Learning for 5G Routing under Side Constraints](https://mlsanity.com/paper/2602.00035/) (2026)
- [MagicSim: A Unified Infrastructure for Executable Embodied Interaction](https://mlsanity.com/paper/2606.17511/) (2026)
- [Conduit: An Experience Data Plane for Distributed Reinforcement Learning](https://mlsanity.com/paper/2609.24456/) (2026)

## Abstract (arXiv)

Researchers and practitioners in the field of reinforcement learning (RL) frequently leverage parallel computation, which has led to a plethora of new algorithms and systems in the last few years. In this paper, we re-examine the challenges posed by distributed RL and try to view it through the lens of an old idea: distributed dataflow. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL, and validate its practicality by porting the full suite of algorithms in RLlib, a widely adopted distributed RL library. Concretely, RLlib Flow provides 2-9 code savings in real production code and enables the composition of multi-agent algorithms not possible by end users before. The open-source code is available as part of RLlib at https://github.com/ray-project/ray/tree/master/rllib.