mlsanityData to 7 Oct 2026

Milestone-Guided Policy Learning for Long-Horizon Language Agents

arXiv 2605.06078 ↗First posted 7 May 2026cs.CL cs.AIcited by 12

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

While long-horizon agentic tasks require language agents to perform dozens of sequential decisions, training such agents with reinforcement learning remains challenging. We identify two root causes: credit misattribution, where correct early actions are penalized due to terminal failures, and sample inefficiency, where scarce successful trajectories result in near-total loss of learning signal. We introduce a milestone-guided policy learning framework, BEACON, that leverages the compositional structure of long-horizon tasks to ensure precise credit assignment. BEACON partitions trajectories at milestone boundaries, applies temporal reward shaping within segments to credit partial progress, and estimates advantages at dual scales to prevent distant failures from corrupting the evaluation of local actions. On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO. Notably, on long-horizon ALFWorld tasks, BEACON achieves 92.9% success rate, nearly doubling GRPO's 53.5%, while improving effective sample utilization from 23.7% to 82.0%. These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents. Code is available at https://github.com/ZJU-REAL/BEACON.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 34 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

25 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
11 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2025Process Reinforcement through Implicit Rewardscited by 478
  5. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  6. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  7. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  8. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  9. 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 56
  10. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 9
  11. 2025GroundAct: Can LLM Agents Ground Actions in Environmental States?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 7
  2. 2026ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RLcited by 3
  3. 2026SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolutioncited by 3
  4. 2026Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirementscited by 2
  5. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1
  6. 2026When to Re-Commit: Temporal Abstraction Discovery for Long-Horizon Vision-Language Reasoningcited by 0
  7. 2026Trajectories That Segment Themselves: Agent-Declared Boundaries as a Training Unitcited by 0
  8. 2026Agent-G2: Gaussian Guidance for Agentic Reinforcement Learningcited by 0
  9. 2026When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluationcited by 0
  10. 2026Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Designcited by 0
  11. 2026Learning to Act with Task Progress: Distilling Small Agents from Compact Teacher Supervisioncited by 0
  12. 2026ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skillscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
  2. 2026MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents
  3. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
  4. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training
  5. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
  6. 2026Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching
  7. 2026Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents
  8. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents