# SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent

arXiv: https://arxiv.org/abs/2511.16108 · HTML: https://mlsanity.com/paper/2511.16108/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 39 of its 56 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 39)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)

## Built on by (most cited first, 30 of 48)

- [Measuring Agents in Production](https://mlsanity.com/paper/2512.04123/) (2025)
- [SWE-World: Building Software Engineering Agents in Docker-Free Environments](https://mlsanity.com/paper/2602.03419/) (2026)
- [SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training](https://mlsanity.com/paper/2602.03411/) (2026)
- [ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents](https://mlsanity.com/paper/2603.18815/) (2026)
- [WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks](https://mlsanity.com/paper/2601.02439/) (2026)
- [ECHO: Terminal Agents Learn World Models for Free](https://mlsanity.com/paper/2605.24517/) (2026)
- [On Randomness in Agentic Evals](https://mlsanity.com/paper/2602.07150/) (2026)
- [ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System](https://mlsanity.com/paper/2602.13692/) (2026)
- [SERA: Soft-Verified Efficient Repository Agents](https://mlsanity.com/paper/2601.20789/) (2026)
- [Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agents](https://mlsanity.com/paper/2606.16038/) (2026)
- [TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents](https://mlsanity.com/paper/2602.11767/) (2026)
- [Revisiting DAgger in the Era of LLM-Agents](https://mlsanity.com/paper/2605.12913/) (2026)
- [Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference](https://mlsanity.com/paper/2605.14220/) (2026)
- [From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents](https://mlsanity.com/paper/2604.01496/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control](https://mlsanity.com/paper/2605.17862/) (2026)
- [APPO: Agentic Procedural Policy Optimization](https://mlsanity.com/paper/2606.12384/) (2026)
- [LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents](https://mlsanity.com/paper/2608.17393/) (2026)
- [ProAct: Agentic Lookahead in Interactive Environments](https://mlsanity.com/paper/2602.05327/) (2026)
- [Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents](https://mlsanity.com/paper/2605.20061/) (2026)
- [Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL](https://mlsanity.com/paper/2606.14211/) (2026)
- [ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.15660/) (2026)
- [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026)
- [Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.09578/) (2026)
- [RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms](https://mlsanity.com/paper/2603.05026/) (2026)
- [LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure](https://mlsanity.com/paper/2606.15306/) (2026)
- [Environment Evolution for Terminal Agents](https://mlsanity.com/paper/2609.04128/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents](https://mlsanity.com/paper/2604.07789/) (2026)

## Abstract (arXiv)

We introduce SkyRL-Agent, a framework for efficient, multi-turn, long-horizon agent training and evaluation. It provides efficient asynchronous dispatching, lightweight tool integration, and flexible backend interoperability, enabling seamless use with existing RL frameworks such as SkyRL-train, VeRL, and Tinker. Using SkyRL-Agent, we train SA-SWE-32B, a software engineering agent trained from Qwen3-32B (24.4% Pass@1) purely with reinforcement learning. We introduce two key components: an optimized asynchronous pipeline dispatcher that achieves a 1.55x speedup over naive asynchronous batching, and a tool-enhanced training recipe leveraging an AST-based search tool to facilitate code navigation, boost rollout Pass@K, and improve training efficiency. Together, these optimizations enable SA-SWE-32B to reach 39.4% Pass@1 on SWE-Bench Verified with more than 2x cost reduction compared to prior models reaching similar performance. Despite being trained solely on SWE tasks, SA-SWE-32B generalizes effectively to other agentic tasks, including Terminal-Bench, BrowseComp-Plus, and WebArena. We further demonstrate SkyRL-Agent's extensibility through case studies on deep research, computer use, and memory agents, each trained using a different training backend.