mlsanityData to 14 Oct 2026

AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework

arXiv 2510.04206 ↗First posted 5 Oct 2025cs.AIcited by 62

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent advances in large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions. However, applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms. In this work, we present the AgentRL framework for scalable multi-turn, multi-task agentic RL training. On the infrastructure side, AgentRL features a fully-asynchronous generation-training pipeline for efficient multi-turn RL. To support heterogeneous environment development in multi-task RL, we design a unified function-call based API interface, containerized environment development, and a centralized controller. On the algorithm side, we propose cross-policy sampling to encourage model exploration in multi-turn settings and task advantage normalization to stabilize multi-task training. Experiments show that AgentRL, trained on open LLMs across five agentic tasks, significantly outperforms GPT-5, Clause-Sonnet-4, DeepSeek-R1, and other open-source LLM agents. Multi-task training with AgentRL matches the best results among all task-specific models. AgentRL is open-sourced at https://github.com/THUDM/AgentRL. The algorithm and framework are adopted in building textschrefhttps://autoglm.zhipuai.cnAutoGLM.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 46 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

34 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  3. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  4. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
18 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  7. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  8. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  9. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  10. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  11. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  13. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 107
  14. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  15. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
  16. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 55
  17. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  18. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 56.

62 papers.

Sort:

By area:

  1. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  2. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  3. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  4. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  5. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 17
  6. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  7. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 13
  8. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 13
  9. 2026RTMC: Step-Level Credit Assignment via Rollout Treescited by 12
  10. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  11. 2025CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RLcited by 10
  12. 2025Klear-AgentForge: Forging Agentic Intelligence through Posttraining Scalingcited by 9
38 more
  1. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rolloutscited by 9
  2. 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
  3. 2025Audited Skill-Graph Self-Improvement for Agentic LLMs via Verifiable Rewards, Experience Synthesis, and Continual Memorycited by 8
  4. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  5. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  6. 2026SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RLcited by 7
  7. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agentscited by 6
  8. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  9. 2026Look Before You Leap: Autonomous Exploration for LLM Agentscited by 6
  10. 2026OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agentscited by 6
  11. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  12. 2026Agentic Tool Use in Large Language Modelscited by 4
  13. 2026OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agentscited by 4
  14. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 4
  15. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  16. 2025SmartSnap: Proactive Evidence Seeking for Self-Verifying Agentscited by 3
  17. 2026ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Mergingcited by 3
  18. 2026TRACE: Capability-Targeted Agentic Trainingcited by 3
  19. 2026Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learningcited by 2
  20. 2026SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Modelscited by 2
  21. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  22. 2026GIFT: Games as Informal Training for Generalizable LLMscited by 1
  23. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMscited by 1
  24. 2026AgentJet: A Distributed Swarm Training Framework for Agentic Reinforcement Learningcited by 1
  25. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  26. 2026OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillationcited by 1
  27. 2026PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Languagecited by 1
  28. 2026From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuationcited by 1
  29. 2026Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agentscited by 1
  30. 2026Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Trainingcited by 1
  31. 2025Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channelscited by 0
  32. 2026AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learningcited by 0
  33. 2026GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agentscited by 0
  34. 2026Trading Human Curation for Synthetic Augmentation in RLVRcited by 0
  35. 2026AgenticRL: Agentic Reinforcement Learning with Self-Refinement for Complex UAV Navigationcited by 0
  36. 2026Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learningcited by 0
  37. 2026MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agentscited by 0
  38. 2026Generating Biomedical Fact-Checking Reports with RL-Enhanced Agentic Searchcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  2. 2025AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
  4. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
  5. 2025Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning
  6. 2025EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models
  7. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
  8. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent