mlsanityData to 14 Oct 2026

Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning

arXiv 2511.14460 ↗First posted 18 Nov 2025cs.CLcited by 29

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language models (LLMs) have rapidly evolved from single-turn text generators into the foundation of increasingly capable agents. As these agents take on more complex reasoning, decision making, tool use, and long-horizon tasks, reinforcement learning (RL) is becoming increasingly important for shaping their behavior. This shift is especially visible in agentic RL, where models must interact with tools and environments across multiple rounds rather than produce a single standalone response. In this regime, the usual view of a trajectory as one ever-growing token sequence becomes increasingly inadequate: it makes context evolution rigid and creates representation mismatches between rollout and training. This paper presents Agent-R1, a unified and modular framework for agentic RL built around step-level trajectory representation, flexible context management, and layered interfaces for workflows, environments and optimization. The key idea is to treat each interaction step as the basic reinforcement-learning transition, while keeping the optimization layer flexible: once the interaction is modeled at the step level, the framework can support token-level credit assignment, step-level credit assignment, or other compatible designs. These design choices make the framework compatible with a range of optimization strategies rather than tying it to a single algorithm. Together, these components provide a principled, extensible, and reusable substrate for agentic RL.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 39 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

31 of 39 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  2. 2025Tree Search for LLM Agent Reinforcement Learning25 Sep
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep
  4. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use1 Sep
  5. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  6. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  8. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks19 Mar
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
10 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  6. 2025Group Sequence Policy Optimizationcited by 835
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  8. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  9. 2024Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Studycited by 8
  10. 2025Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 28.

29 papers.

Sort:

By area:

  1. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  2. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  3. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 12
  4. 2026Cast-R1: Learning Tool-Augmented Sequential Decision Policies for Time Series Forecastingcited by 10
  5. 2026Position: Beyond Model-Centric Prediction -- Agentic Time Series Forecastingcited by 9
  6. 2026WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoningcited by 8
  7. 2026EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environmentscited by 8
  8. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1cited by 8
  9. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  10. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  11. 2025Toward a Safe Internet of Agentscited by 6
  12. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Searchcited by 6
17 more
  1. 2026EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repaircited by 6
  2. 2026PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimizationcited by 5
  3. 2026AnomaMind: Agentic Time Series Anomaly Detection with Tool-Augmented Reasoningcited by 5
  4. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  5. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspectivecited by 4
  6. 2026AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraintscited by 2
  7. 2026SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Modelscited by 2
  8. 2026Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysiscited by 2
  9. 2026Tools as Continuous Flow for Evolving Agentic Reasoningcited by 2
  10. 2026MetaAgent-X : Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learningcited by 2
  11. 2026ContractBench: Can LLM Agents Preserve Observation Contracts?cited by 2
  12. 2026TableMind++: An Uncertainty-Aware Programmatic Agent for Tool-Augmented Table Reasoningcited by 1
  13. 2026Internalizing Agency from Reflective Experiencecited by 1
  14. 2026AIT Academy: Cultivating the Complete Agent with a Confucian Three-Domain Curriculumcited by 0
  15. 2026Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoningcited by 0
  16. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Modelscited by 0
  17. 2026Cowork-Gym: Synthesizing Interactive Work Environments at Scalecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  2. 2025Demystifying Reinforcement Learning in Agentic Reasoning
  3. 2025AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents
  4. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
  5. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
  6. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  7. 2025Agent2: An Agent-Generates-Agent Framework for Reinforcement Learning Automation
  8. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization