mlsanityData to 14 Oct 2026

MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents

arXiv 2608.19803 ↗First posted 20 Aug 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Credit assignment is challenging in long-horizon agentic reinforcement learning, where supervision often comes only from final rewards. Existing methods refine trajectory-level signals into step-level credits through step grouping or graph-based advantage estimation, but can overlook meaningful intermediate milestones. We propose MileGPO (Milestone Inference with Local Evidence for Graph-Based Policy Optimization), which derives process-level credit from grouped on-policy rollouts through three designs. Milestone Discovery identifies candidate milestones on successful rollouts and recurring traps on failed ones. Reliability-Calibrated Shaping (RCS) weights these candidates by outcome-based confidence, strengthening reliable milestones and traps while down-weighting uncertain ones. Progress-Contrastive Calibration (PCC) further tests whether a candidate reflects local progress and whether its incoming transition outperforms observed alternatives from the same state. MileGPO requires neither auxiliary models nor additional environment interaction. Experiments on ALFWorld and WebShop show state-of-the-art performance and a small in-distribution to out-of-distribution gap on ALFWorld. Ablations and credit diagnostics indicate that reliability weighting, local progress, and same-state branch evidence complement milestone discovery and resolve ambiguous intermediate credit.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 44 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

28 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents24 Jun
  2. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents24 Jun
  3. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning22 Jun
  4. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning26 May
  5. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  6. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  7. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
7 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  3. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  4. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  5. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  6. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  7. 2026GraphPO: Graph-based Policy Optimization for Reasoning Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agentscited by 0
  2. 2026Matched-Context Group Policy Optimization for LLM Agent Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  2. 2026Learning from Viable Failure Prefixes: Milestone Viability Potential Policy Optimization for Long-Horizon LLM Agents
  3. 2026Milestone-Guided Policy Learning for Long-Horizon Language Agents
  4. 2026When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
  5. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  6. 2026AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
  7. 2026GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
  8. 2026Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents