mlsanityData to 14 Oct 2026

EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning

arXiv 2509.22576 ↗First posted 26 Sep 2025cs.LG cs.CLcited by 17

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Training LLM agents in multi-turn environments with sparse rewards, where completing a single task requires 30+ turns of interaction within an episode, presents a fundamental challenge for reinforcement learning. We identify a critical failure mode unique to this setting: the exploration-exploitation cascade failure. This cascade begins with early-stage policy premature convergence, where sparse feedback causes agents to commit to flawed, low-entropy strategies. Subsequently, agents enter late-stage policy collapse, where conventional entropy regularization becomes counterproductive, promoting chaotic exploration that destabilizes training. We propose Entropy-regularized Policy Optimization (EPO), a general framework that breaks this failure cycle through three synergistic mechanisms: (1) adopting entropy regularization in multi-turn settings to enhance exploration, (2) an entropy smoothing regularizer that bounds policy entropy within historical averages to prevent abrupt fluctuations, and (3) adaptive phase-based weighting that balances exploration and exploitation across training. Our analysis justifies that EPO guarantees monotonically decreasing entropy variance while maintaining convergence. EPO achieves up to 152% performance improvement on ScienceWorld and up to 19.8% on ALFWorld. Our work demonstrates that multi-turn sparse-reward settings require fundamentally different entropy control than traditional RL, with broad implications for LLM agent training.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 45 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

35 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
20 more
  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2025Kimi K2: Open Agentic Intelligencecited by 567
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  8. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  9. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  10. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  11. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  12. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  13. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  14. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  15. 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 58
  16. 2025On Entropy Control in LLM-RL Algorithmscited by 40
  17. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  18. 2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
  19. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVRcited by 17
  20. 2023Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methodscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 20
  2. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 16
  3. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
  4. 2026AEL: Evolving Agent Harness in Open-Ended Environmentscited by 6
  5. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learningcited by 5
  6. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoningcited by 3
  7. 2026When Does Multi-Agent Collaboration Help? An Entropy Perspectivecited by 3
  8. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  9. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  10. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Controlcited by 1
  11. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimizationcited by 1
  12. 2026Cyclical Entropy Eruption: Entropy Dynamics in Agent Reinforcement Learningcited by 1
5 more
  1. 2026STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Trainingcited by 1
  2. 2026When to Re-Commit: Temporal Abstraction Discovery for Long-Horizon Vision-Language Reasoningcited by 0
  3. 2026S3: Spectral Null-Space Swap Makes Reasoning Models Efficientcited by 0
  4. 2026RELACE: retrospective likelihood-based action credit estimation for long-horizon language agentscited by 0
  5. 2026TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  2. 2026Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning
  3. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
  4. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  5. 2026Entropy-Preserving Reinforcement Learning
  6. 2026FERPO: Forward Entropy-Regularized Policy Optimization
  7. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
  8. 2026Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization