mlsanityData to 7 Oct 2026

Agentic Entropy-Balanced Policy Optimization

arXiv 2510.14545 ↗First posted 16 Oct 2025cs.LG cs.AI cs.CLcited by 39

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable web agent training.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 66 of its 101 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

66 of 101 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Tree Search for LLM Agent Reinforcement Learning25 Sep
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep
  3. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  4. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use1 Sep
  5. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  7. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  8. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 57

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
38 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  4. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  5. 2025Group Sequence Policy Optimizationcited by 788
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  10. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  11. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  12. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  13. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  14. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  15. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  16. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  17. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  18. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  19. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  20. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  21. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  22. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 132
  23. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  24. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 118
  25. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  26. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  27. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 70
  28. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  29. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 66
  30. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 60
  31. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
  32. 2025EvolveSearch: An Iterative Self-Evolving Search Agentcited by 42
  33. 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Trainingcited by 42
  34. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  35. 2025Scaling Agents via Continual Pre-trainingcited by 39
  36. 2024HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systemscited by 37
  37. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  38. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 37.

39 papers.

Sort:

By area:

  1. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  2. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  3. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  4. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  5. 2025SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learningcited by 6
  6. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimizationcited by 6
  7. 2026E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoningcited by 6
  8. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
  9. 2026Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agentscited by 5
  10. 2026Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoningcited by 5
  11. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  12. 2026Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimizationcited by 5
27 more
  1. 2026ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibrationcited by 4
  2. 2026CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learningcited by 4
  3. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  4. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 4
  5. 2026From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Searchcited by 4
  6. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  7. 2026APPO: Agentic Procedural Policy Optimizationcited by 3
  8. 2026Agent Explorative Policy Optimization for Multimodal Agentic Reasoningcited by 2
  9. 2026Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agentscited by 2
  10. 2026Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learningcited by 2
  11. 2026AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizingcited by 2
  12. 2026Co-Evolution of Policy and Internal Reward for Language Agentscited by 1
  13. 2026T2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learningcited by 1
  14. 2026Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learningcited by 1
  15. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  16. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capabilitycited by 1
  17. 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 1
  18. 2025GPG: Generalized Policy Gradient Theorem for Transformer-based Policiescited by 0
  19. 2026Dep-Search: Learning Dependency-Aware Reasoning Traces with Persistent Memorycited by 0
  20. 2026Differentiable Mixture-of-Agents Incentivizes Swarm Intelligence of Large Language Modelscited by 0
  21. 2026ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learningcited by 0
  22. 2026RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agentscited by 0
  23. 2026Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learningcited by 0
  24. 2026From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasonerscited by 0
  25. 2026GenMem: Generative Symbolic Memory for Self-Evolving Harnesscited by 0
  26. 2026SIPO: Selective-Inference Policy Optimization for Tree-Structured Agentic RLcited by 0
  27. 2026When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning
  2. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature
  3. 2026Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning
  4. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
  5. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
  6. 2025Agentic Reinforced Policy Optimization
  7. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Search
  8. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization