# Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

arXiv: https://arxiv.org/abs/2605.27788 · HTML: https://mlsanity.com/paper/2605.27788/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://mlsanity.com/paper/2307.04964/) (2023)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.19470/) (2025)
- [Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models](https://mlsanity.com/paper/2505.23564/) (2025)
- [R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.23794/) (2025)
- [Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning](https://mlsanity.com/paper/2507.01489/) (2025)
- [Execution-Grounded Credit Assignment for GRPO in Code Generation](https://mlsanity.com/paper/2603.16158/) (2026)
- [Learning to Configure Agentic AI Systems](https://mlsanity.com/paper/2602.11574/) (2026)

## Built on by (most cited first, 3 of 3)

- [SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents](https://mlsanity.com/paper/2608.18852/) (2026)
- [Trajectories That Segment Themselves: Agent-Declared Boundaries as a Training Unit](https://mlsanity.com/paper/2608.02302/) (2026)
- [Auditing Future-Memory Use in Point-in-Time Language Agents](https://mlsanity.com/paper/2610.18602/) (2026)

## Abstract (arXiv)

Humans know when to reach for help e.g. 347 × 28 warrants a calculator while 2+2 does not. Language models do not. Prompt-based approaches can instruct a model when to invoke tools, but this scaffolding does not teach it to recognize the boundary of its own knowledge. RL approaches that assign a single outcome reward to the whole trajectory fare no better: trajectory-level credit cannot isolate which tool call in a successful episode actually helped, nor penalize unnecessary calls. We propose CARL (Competence-Aware Reinforcement Learning), which trains a critic on the model's own rollouts to learn where parametric knowledge suffices and where it needs external help. By decomposing each rollout at natural tool-use boundaries (e.g., code fence delimiters and context block transitions), CARL assigns independent credit to each segment from a single binary outcome, without external judges or step-level annotations. As a result, erroneous tool calls, incorrect extractions, and unnecessary calls each receive appropriately signed advantages. The trained critic captures the model's domain competence: it separates parametrically solvable from tool-dependent questions with AUC 0.93 at 7B. On five benchmarks spanning arithmetic, multi-hop factual QA, and numerical reasoning over financial tables, CARL improves exact-match accuracy by 6.7 points at 7B and 9.7 points at 3B over the best RL baseline, with the largest gain (+8.3 EM at 7B, +9.0 EM at 3B) on Musique. The model issues 53% fewer tool calls on parametrically answerable questions while remaining ~10 EM points more accurate on them. Gains are largest at small scale: the 3B improvement is 1.4× the 7B improvement, suggesting that knowing when to ask disproportionately benefits models with smaller parametric memory.