mlsanityData to 14 Oct 2026

StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation

arXiv 2504.15930 ↗First posted 22 Apr 2025cs.LG cs.DCcited by 61

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs). RL for LLMs involves two stages: generation and training. The LLM first generates samples online, which are then used to derive rewards for training. The conventional view holds that the colocated architecture, where the two stages share resources via temporal multiplexing, outperforms the disaggregated architecture, in which dedicated resources are assigned to each stage. However, in real-world deployments, we observe that the colocated architecture suffers from resource coupling, where the two stages are constrained to use the same resources. This coupling compromises the scalability and cost-efficiency of colocated RL in large-scale training. In contrast, the disaggregated architecture allows for flexible resource allocation, supports heterogeneous training setups, and facilitates cross-datacenter deployment. StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching and scheduling. Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems, and improves cost-effectiveness by up to 1.33x in a heterogeneous, cross-datacenter setting.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

28 of 46 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
16 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  6. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  7. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  8. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  9. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  10. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  11. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  12. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  13. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  14. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  15. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  16. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 42.

61 papers.

Sort:

By area:

  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  2. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  3. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  4. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  5. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  6. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  7. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  8. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  9. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  10. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 14
  11. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 14
  12. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
38 more
  1. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 11
  2. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
  3. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 10
  4. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 10
  5. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  6. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  7. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  8. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  9. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learningcited by 7
  10. 2025SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Schedulingcited by 4
  11. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  12. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  13. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  14. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 4
  15. 2026FlashEvolve: Accelerating Agent Self-Evolution with Asynchronous Stage Orchestrationcited by 3
  16. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  17. 2026AIS: Adaptive Importance Sampling for Quantized RLcited by 3
  18. 2026AsyncOPD: How Stale Can On-Policy Distillation Be?cited by 3
  19. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learningcited by 3
  20. 2026ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learningcited by 2
  21. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  22. 2026DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learningcited by 2
  23. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  24. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 2
  25. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  26. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  27. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  28. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1
  29. 2026MinT: Managed Infrastructure for Training and Serving Millions of LLMscited by 1
  30. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMscited by 1
  31. 2026Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelismcited by 1
  32. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  33. 2026Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Trainingcited by 1
  34. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 1
  35. 2026ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learningcited by 0
  36. 2026Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Trainingcited by 0
  37. 2026Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generationcited by 0
  38. 2026Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learningcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  2. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
  3. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
  4. 2026PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
  5. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
  6. 2025Laminar: A Scalable Asynchronous RL Post-Training Framework
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  8. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning