mlsanityData to 7 Oct 2026

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

arXiv 2609.12424 ↗First posted 11 Sep 2026cs.LGcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Long-horizon language-model agents trained with reinforcement learning oftenreceive sparse outcome rewards that do not reveal which decisions along a tra-jectory deserve credit. Episode-level advantages provide coarse trajectory-widecredit, while step-level comparisons offer finer resolution with context-dependentestimation noise. We propose Granularity-Adaptive Credit Assignment (GACA),a critic-free method that adaptively mixes episode- and step-level credit for eachdecision during policy optimization. GACA normalizes the sampled response'smean per-token negative log-likelihood (NLL) within each trajectory and uses theresulting criticality score to determine the step-specific mixture. The computationreuses rollout log-probabilities without additional training rollouts or model eval-uations. Our analysis characterizes optimal score-dependent mixing and derivesconditions linking expected NLL to a lower bound on the preferred step-levelweight. Across ALFWorld and WebShop with 1.5B and 7B backbones, GACAachieves the highest reported mean success rates among the compared methods,while introducing negligible additional computation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 32 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

27 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks22 Jul
  2. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents24 Jun
  3. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning1 May
  4. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  5. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
9 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  6. 2018An Introduction to Deep Reinforcement Learningcited by 201
  7. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  8. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  9. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EvoSteer: Online Self-Evolving Graph Orchestration via Reference-Anchored Credit Assignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
  2. 2026Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
  3. 2026When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
  4. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  5. 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
  6. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
  7. 2026PACT: From Credit Assignment to Critic Alignment
  8. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment