# ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents

arXiv: https://arxiv.org/abs/2603.18815 · HTML: https://mlsanity.com/paper/2603.18815/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 25 of 25)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [Free Process Rewards without Process Labels](https://mlsanity.com/paper/2412.01981/) (2024)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [Agent Lightning: Train ANY AI Agents with Reinforcement Learning](https://mlsanity.com/paper/2508.03680/) (2025)
- [VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use](https://mlsanity.com/paper/2509.01055/) (2025)
- [Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning](https://mlsanity.com/paper/2505.00024/) (2025)
- [VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents](https://mlsanity.com/paper/2510.16907/) (2025)
- [Offline Training of Language Model Agents with Functions as Learnable Weights](https://mlsanity.com/paper/2402.11359/) (2024)
- [SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent](https://mlsanity.com/paper/2511.16108/) (2025)
- [SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain](https://mlsanity.com/paper/2501.15587/) (2025)
- [GEM: A Gym for Agentic LLMs](https://mlsanity.com/paper/2510.01051/) (2025)

## Built on by (most cited first, 20 of 20)

- [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026)
- [Orchard: An Open-Source Agentic Modeling Framework](https://mlsanity.com/paper/2605.15040/) (2026)
- [Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale](https://mlsanity.com/paper/2604.11554/) (2026)
- [Revisiting DAgger in the Era of LLM-Agents](https://mlsanity.com/paper/2605.12913/) (2026)
- [Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning](https://mlsanity.com/paper/2608.09682/) (2026)
- [DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training](https://mlsanity.com/paper/2607.22614/) (2026)
- [Learning CLI Agents with Structured Action Credit under Selective Observation](https://mlsanity.com/paper/2605.08013/) (2026)
- [A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions](https://mlsanity.com/paper/2606.16733/) (2026)
- [ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination](https://mlsanity.com/paper/2609.24526/) (2026)
- [ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoning](https://mlsanity.com/paper/2605.18799/) (2026)
- [When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR](https://mlsanity.com/paper/2605.19425/) (2026)
- [Trading Human Curation for Synthetic Augmentation in RLVR](https://mlsanity.com/paper/2606.03800/) (2026)
- [Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.07178/) (2026)
- [JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models](https://mlsanity.com/paper/2607.16074/) (2026)
- [Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.21653/) (2026)
- [Memory Compression for High-Fanout Agent Sandboxes](https://mlsanity.com/paper/2609.11294/) (2026)
- [Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions](https://mlsanity.com/paper/2609.25463/) (2026)
- [SCLATE: a Substrate for Continual-Learning Agent Training and Evaluation](https://mlsanity.com/paper/2609.32391/) (2026)
- [LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches](https://mlsanity.com/paper/2610.06647/) (2026)
- [ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs](https://mlsanity.com/paper/2610.15368/) (2026)

## Abstract (arXiv)

Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.