mlsanityData to 14 Oct 2026

Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning

arXiv 2605.26684 ↗First posted 26 May 2026cs.LG cs.AIcited by 31

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs) and have been rapidly extended to agentic tasks. However, their credit assignment relies heavily on coarse-grained trajectory-level attribution according to final outcomes, making it difficult to capture the contribution of individual steps, such as valuable steps obscured within failed trajectories. To uncover latent information and enable more faithful step-level credit assignment, we propose Graph-based Group Policy Optimization (GraphGPO), which first aggregates all rollout trajectories into a unified state-transition graph and then estimates the distance from each state to the task goal using the global information encoded in the graph. Finally, GraphGPO assigns credit to each edge by estimating a graph-based advantage, based on how much the transition reduces the distance to the task goal. In this way, GraphGPO significantly improves training efficiency and achieves state-of-the-art performance across a range of challenging benchmarks.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 81 of its 93 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

81 of 93 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  2. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents27 Oct
  3. 2025ToolBrain: A Flexible Reinforcement Learning Framework for Agentic Tools24 Sep
  4. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  5. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  6. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  7. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards13 Jun
  8. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  9. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning1 May
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  11. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently21 Apr
  12. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
4 more
  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  2. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game29 Oct
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 65

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
38 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  13. 2016OpenAI Gymcited by 1,736
  14. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  15. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  16. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  17. 2023Mastering Diverse Domains through World Modelscited by 1,253
  18. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  19. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  20. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  21. 2025Group Sequence Policy Optimizationcited by 835
  22. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  23. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  24. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  25. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  26. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  27. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  28. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  29. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  30. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  31. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  32. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  33. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  34. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  35. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  36. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  37. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  38. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 31.

31 papers.

Sort:

By area:

  1. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 9
  2. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agentscited by 7
  3. 2026SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learningcited by 3
  4. 2026Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agentscited by 2
  5. 2026DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agentscited by 2
  6. 2026MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agentscited by 2
  7. 2026When Does Muon Help Agentic Reinforcement Learning?cited by 1
  8. 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
  9. 2026Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learningcited by 1
  10. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agentscited by 1
  11. 2026Groupwise Agentic Grading and Advantage Redistribution for Code Agent RLcited by 1
  12. 2026Back to the Definition: Estimating Step-Level Advantages via Trajectory Graphs for Agentic Reinforcement Learningcited by 0
19 more
  1. 2026From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMscited by 0
  2. 2026Multi-Agent System Search via Active Substructure-aware Policy Optimizationcited by 0
  3. 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agentscited by 0
  4. 2026UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learningcited by 0
  5. 2026TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RLcited by 0
  6. 2026Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learningcited by 0
  7. 2026GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 0
  8. 2026Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teacherscited by 0
  9. 2026PRICE the Action Chunks: Physical Relational Credit Assignment for Embodied Reinforcement Learningcited by 0
  10. 2026It Takes Workflows to Evolve Better Workflowscited by 0
  11. 2026Dependency-Aware Reward Shaping for Agentic Reinforcement Learningcited by 0
  12. 2026Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agentscited by 0
  13. 2026RELACE: retrospective likelihood-based action credit estimation for long-horizon language agentscited by 0
  14. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agentscited by 0
  15. 2026When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimationcited by 0
  16. 2026SynAct: Synergistic Active Feedback Learning for LLM Agentscited by 0
  17. 2026Matched-Context Group Policy Optimization for LLM Agent Trainingcited by 0
  18. 2026VH-GPO: Verified Handoff for LLM Agent Reinforcement Learningcited by 0
  19. 2026What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
  2. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  3. 2026SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
  4. 2026Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
  5. 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
  6. 2026PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
  7. 2026RTMC: Step-Level Credit Assignment via Rollout Trees
  8. 2026Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization