mlsanityData to 14 Oct 2026

RollArt: Disaggregated Multi-Task Agentic RL Training at Scale

arXiv 2512.22560 ↗First posted 27 Dec 2025cs.DC cs.AI cs.LGcited by 14

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Agentic Reinforcement Learning (RL) trains LLMs through multi-turn interactions with environments, producing workloads that mix compute-bound prefill, bandwidth-bound decoding, CPU-heavy environment execution, and bursty reward evaluation. Existing systems either colocate all stages on a single GPU cluster or decouple them only at a coarse granularity, overlooking hardware heterogeneity and incurring substantial synchronization overhead across stages. We present ROLLART, a system for multi-task agentic RL on disaggregated infrastructure. ROLLART maps each pipeline stage to best-fit hardware, routing prefill-heavy tasks to compute-optimized GPUs, decode-heavy tasks to bandwidth-optimized GPUs, and environments to CPU clusters. It decouples rollout at the trajectory level, allowing generation, environment interaction, and reward scoring to proceed independently, so that slow or failed environments never block the others. ROLLART offloads stateless reward computation to serverless infrastructure and overlaps rollout with training via staleness-bounded asynchronous weight synchronization. Our results demonstrate that ROLLART effectively improves training throughput and achieves 1.31--2.05 (×) training time reduction compared to various RL systems. We also evaluated ROLLART by training a hundreds-of-billions-parameter MoE model for Qoder product on an Alibaba cluster with above 3,000 GPUs, demonstrating its stability and scalability.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 63 references that are in the map. First posted on arXiv 27 Dec 2025.

Builds on

42 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony13 Oct
  2. 2025AWorld: Orchestrating the Training Recipe for Agentic AI28 Aug

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
28 more
  1. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  2. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  3. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  4. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  5. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  6. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  7. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  8. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  10. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 121
  11. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  12. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  13. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  14. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  15. 2025DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learningcited by 66
  16. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  17. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  18. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  19. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  20. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  21. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  22. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  23. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 25
  24. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  25. 2023Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inferencecited by 13
  26. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 10
  27. 2025SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Schedulingcited by 4
  28. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 30
  2. 2026Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxescited by 13
  3. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  4. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  5. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  6. 2026D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Modelscited by 1
  7. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 1
  8. 2026Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Trainingcited by 0
  9. 2026Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generationcited by 0
  10. 2026Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learningcited by 0
  11. 2026Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learningcited by 0
  12. 2026PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learningcited by 0
2 more
  1. 2026Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learningcited by 0
  2. 2026Efficient Resource-Shiftable RL Post-training with Generative Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
  2. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
  3. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  4. 2026Belayer: Efficient Fault Tolerance for LLM Agentic RL Training
  5. 2026Heddle: A Distributed Orchestration System for Agentic RL Rollout
  6. 2026EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management
  7. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  8. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs