mlsanityData to 14 Oct 2026

Agentic Reinforced Policy Optimization

arXiv 2507.19849 ↗First posted 26 Jul 2025cs.LG cs.AI cs.CLcited by 172

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks. In realistic reasoning scenarios, LLMs can often utilize external tools to assist in task-solving processes. However, current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents. Through preliminary experiments, we observe that LLMs tend to exhibit highly uncertain behavior, characterized by an increase in the entropy distribution of generated tokens, immediately following interactions with external tools. Motivated by this observation, ARPO incorporates an entropy-based adaptive rollout mechanism, dynamically balancing global trajectory sampling and step-level sampling, thereby promoting exploration at steps with high uncertainty after tool usage. By integrating an advantage attribution estimation, ARPO enables LLMs to internalize advantage differences in stepwise tool-use interactions. Our experiments across 13 challenging benchmarks in computational reasoning, knowledge reasoning, and deep search domains demonstrate ARPO's superiority over trajectory-level RL algorithms. Remarkably, ARPO achieves improved performance using only half of the tool-use budget required by existing methods, offering a scalable solution for aligning LLM-based agents with real-time dynamic environments. Our code and datasets are released at https://github.com/dongguanting/ARPO

Key facts. Cited by 172 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 85 of its 103 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

85 of 103 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025First Return, Entropy-Eliciting Explore9 Jul
  2. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  4. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  5. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  6. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  11. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  12. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning10 Feb
4 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 69

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
38 more
  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  3. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  6. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  7. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  10. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  11. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  12. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  13. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  14. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  15. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  16. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  17. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  18. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  19. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  20. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  21. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  22. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  23. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  24. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  25. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  26. 2025ToRL: Scaling Tool-Integrated RLcited by 169
  27. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  28. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  29. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161
  30. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  31. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  32. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  33. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 110
  34. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
  35. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  36. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  37. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  38. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 45; 2026: 127.

172 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  3. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141
  4. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  5. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  6. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 65
  7. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  8. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  9. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  10. 2025ReasonRank: Empowering Passage Ranking with Strong Reasoning Abilitycited by 42
  11. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  12. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 37
38 more
  1. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  2. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  3. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  4. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  6. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  7. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  8. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 31
  9. 2026Gen-Searcher: Reinforcing Agentic Search for Image Generationcited by 30
  10. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  11. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  12. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 24
  13. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  14. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  15. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  16. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 19
  17. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  18. 2025DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agentscited by 17
  19. 2025Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learningcited by 17
  20. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  21. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  22. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  23. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  24. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 15
  25. 2026ClawGym: A Scalable Framework for Building Effective Claw Agentscited by 15
  26. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  27. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMscited by 14
  28. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
  29. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  30. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
  31. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 12
  32. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 12
  33. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  34. 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 11
  35. 2025GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learningcited by 11
  36. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
  37. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  38. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Modelscited by 11

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  3. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  4. 2025Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization
  5. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  6. 2026AIPO: Learning to Reason from Active Interaction
  7. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  8. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization