mlsanityData to 14 Oct 2026

Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents

arXiv 2610.03634 ↗First posted 2 Oct 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models also fits: Deep RL algorithms and representations

Abstract (arXiv)

Terminal-using agents benefit from reinforcement learning (RL) in coding, debugging, and other multi-step terminal tasks. In these tasks, later commands often depend on information or intermediate results produced by earlier commands. However, existing trajectory-level and step-level credit assignment methods do not explicitly trace the read-write dependencies through which commands affect the final outcome. Consequently, training signals could still be assigned to irrelevant operations, weakening learning from relevant steps. In this paper, we propose Dependency-Aware Group Policy Optimization (DepGPO), which uses execution dependencies between commands to guide credit assignment for terminal agents. Specifically, we construct a command dependency graph from execution traces and trace backward from the resources inspected by the task verifier. We then assign credit to relevant writes and their supporting reads along these paths, and use it to redistribute trajectory advantages across steps. Extensive comparative experiments and ablation studies demonstrate that DepGPO improves task performance and training stability on complex terminal tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 53 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

39 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents29 Sep
  2. 2026T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks10 Sep

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  8. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  9. 2025Process Reinforcement through Implicit Rewardscited by 504
  10. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  12. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 116
25 more
  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  2. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 65
  3. 2026Endless Terminals: Scaling RL Environments for Terminal Agentscited by 54
  4. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  5. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 37
  6. 2026Tmax: A simple recipe for terminal agentscited by 33
  7. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 31
  8. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  9. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  10. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 17
  11. 2026CodeTracer: Towards Traceable Agent Statescited by 16
  12. 2026SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scalingcited by 16
  13. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 13
  14. 2026Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Tracescited by 12
  15. 2026SETA: Scaling Environments for Terminal Agentscited by 12
  16. 2026AgentTrace: A Structured Logging Framework for Agent System Observabilitycited by 9
  17. 2026PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planningcited by 9
  18. 2026GAGPO: Generalized Advantage Grouped Policy Optimizationcited by 9
  19. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 9
  20. 2026Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Makingcited by 8
  21. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learningcited by 6
  22. 2025Graph-Enhanced Policy Optimization in LLM Agent Trainingcited by 5
  23. 2026SWE-Shepherd: Advancing PRMs for Reinforcing Code Agentscited by 5
  24. 2026Learning Generalizable Behaviors for Terminal Agentscited by 2
  25. 2026Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
  2. 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
  3. 2026DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training
  4. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  5. 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
  6. 2026Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents
  7. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
  8. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning