mlsanityData to 14 Oct 2026

When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training

arXiv 2606.05885 ↗First posted 4 Jun 2026cs.LGcited by 7

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards. Recent group-based methods such as GiGPO improve over GRPO by constructing step-level advantages at repeated anchor states. However, we show that such dense credit can be statistically unreliable: under limited rollouts, rare but lucky actions may receive overly large advantages, producing divergent anchor bias and late-stage training oscillation. We propose Evidence-Calibrated Policy Optimization (ECPO), a critic-free policy optimization algorithm that calibrates step-level credit before policy updates. ECPO combines Evidence-Calibrated Action Advantage, which groups rollouts by canonical actions and shrinks low-count estimates, with Variance-Gated Credit Weighting, which suppresses anchor states dominated by within-action noise. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B show that ECPO consistently outperforms strong baselines, improving GiGPO by +5.2/+7.3 success points on ALFWorld/WebShop with Qwen2.5-1.5B while adding only 0.1% additional advantage-computation overhead.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 38 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

24 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Hindsight Credit Assignment for Long-Horizon LLM Agents7 Mar
  2. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization3 Mar
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
8 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  3. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  4. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  5. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 165
  6. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 96
  7. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  8. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 71

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026TRACER: Per-Tool Context Retention for LLM Agents via Consequence-Attributed Reinforcement Learningcited by 1
  2. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  3. 2026CREDO: Variance-Guided Rubric Evolution for Replay-Corrected Credit Assignmentcited by 0
  4. 2026Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learningcited by 0
  5. 2026AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learningcited by 0
  6. 2026RELACE: retrospective likelihood-based action credit estimation for long-horizon language agentscited by 0
  7. 2026Matched-Context Group Policy Optimization for LLM Agent Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from Viable Failure Prefixes: Milestone Viability Potential Policy Optimization for Long-Horizon LLM Agents
  2. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  3. 2026MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents
  4. 2026Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning
  5. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  6. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
  7. 2026Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
  8. 2026PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks