SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
On the map: LLM reasoning → Reinforcement learning for reasoning
also in 1 more
Abstract (arXiv)
Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens. Consequently, gradient noise from summary generation leaks into tool-decision tokens, causing cross-segment credit misattribution and brittle optimization. In this work, we propose SLCA-GRPO, a framework incorporating Segment-Locked Credit Assignment (SLCA). To enable scalable exploration without costly real APIs and stable training, we first construct the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure. Building on this, SLCA decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), SLCA routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination (the dominant cross-segment credit misattribution channel) within each policy update. On a 7B backbone, SLCA-GRPO accelerates convergence and outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on τ2-Bench under the same training budgets, achieving higher accuracy with reduced tool redundancy and costs.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 42 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
10 of 42 references.
Closest ancestors · 5
Same direction, newest first
- 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards22 Dec
- 2025From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs19 Nov
- 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
- 2022Solving math word problems with process- and outcome-based feedback25 Nov
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 5
- 2025Qwen3 Technical Reportcited by 8,195
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
- 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
- 2026TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents
- 2026SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
- 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
- 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
- 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
- 2026PACT: From Credit Assignment to Critic Alignment
- 2026Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning