mlsanityData to 14 Oct 2026

Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

arXiv 2610.16592 ↗First posted 12 Oct 2026cs.DC cs.LGcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks. During rollout, agentic RL exhibits a workload pattern distinct from that of traditional RL: multi-turn interactions repeatedly introduce prefill requests, causing prefill and decode workloads to coexist throughout rollout. This workload makes prefill-decode (PD) disaggregation, a design widely used for online LLM serving, promising for speeding up agentic RL rollout. This paper shows that PD disaggregation can improve rollout efficiency by (a) mitigating interference among data-parallel attention modules, (b) enabling stage-specific parallelism, and (c) reducing end-to-end trajectory completion time. However, a fixed prefill-decode allocation may fail to sustain these benefits as workload composition varies across tasks and over time. We therefore present Prorsa, an agentic RL framework for prediction-guided prefill-decode resource allocation. Prorsa observes the lifecycle of every trajectory to forecast short-horizon prefill and decode demand. These forecasts then guide worker allocation decisions and trigger role switching of rollout engines, aligning prefill and decode capacity with anticipated workload demand. Across synchronous and asynchronous agentic RL settings, Prorsa achieves end-to-end rollout speedups of 1.11x-4.05x over prefill-decode colocation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 60 of its 72 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

60 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning19 Jul
  2. 2026EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents7 Jul
  3. 2026Libra: Efficient Resource Management for Agentic RL Post-Training2 Jun
  4. 2026Heddle: A Distributed Orchestration System for Agentic RL Rollout30 Mar
  5. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scale27 Dec

Other references · 55

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
38 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  2. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  3. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  4. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  6. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  8. 2026Kimi K3: Open Frontier Intelligencecited by 138
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  10. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  11. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 85
  12. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  13. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  14. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 53
  15. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  16. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  17. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  18. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  19. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  20. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  21. 2026APEX-Agentscited by 32
  22. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  23. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  24. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  25. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  26. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 16
  27. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  28. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
  29. 2025DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Trainingcited by 10
  30. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 10
  31. 2026Qwen-CUA: Native Computer Use for (almost) Everythingcited by 8
  32. 2026Efficient Multi-round LLM Inference over Disaggregated Servingcited by 7
  33. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Servingcited by 6
  34. 2026Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Servingcited by 6
  35. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  36. 2026OrchestrRL: Dynamic Compute and Network Orchestration for Disaggregated RLcited by 5
  37. 2025SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Schedulingcited by 4
  38. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning
  2. 2026Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
  3. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  4. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization
  5. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
  6. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
  7. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
  8. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts