Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
also in 1 more
Abstract (arXiv)
How much should a language agent think before taking action? Chain-of-thought (CoT) reasoning is widely assumed to improve agent performance, but the relationship between reasoning length and accuracy in structured tool-use settings remains poorly understood. We present a systematic study of CoT budget effects on function-calling agents, sweeping six token budgets (0--512) across 200 tasks from the Berkeley Function Calling Leaderboard v3 Multiple benchmark. Our central finding is a striking non-monotonic pattern on Qwen2.5-1.5B-Instruct: brief reasoning (32 tokens) dramatically improves accuracy by 45% relative over direct answers, from 44.0% to 64.0%, while extended reasoning (256 tokens) degrades performance well below the no-CoT baseline, to 25.0% (McNemar p < 0.001). A three-way error decomposition reveals the mechanism. At d = 0, 30.5% of tasks fail because the model selects the wrong function from the candidate set; brief CoT reduces this to 1.5%, effectively acting as a function-routing step, while long CoT reverses the gain, yielding 28.0% wrong selections and 18.0% hallucinated functions at d = 256. Oracle analysis shows that 88.6% of solvable tasks require at most 32 reasoning tokens, with an average of 27.6 tokens, and a finer-grained sweep indicates that the true optimum lies at 8--16 tokens. Motivated by this routing effect, we propose Function-Routing CoT (FR-CoT), a structured brief-CoT method that templates the reasoning phase as "Function: [name] / Key args: [...]," forcing commitment to a valid function name at the start of reasoning. FR-CoT achieves accuracy statistically equivalent to free-form d = 32 CoT while reducing function hallucination to 0.0%, providing a structural reliability guarantee without budget tuning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 11 of its 17 references that are in the map. First posted on arXiv 2 Apr 2026.
Builds on
11 of 17 references.
References · 11
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
- 2019Depth-Adaptive Transformercited by 227
- 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
- 2024Sub-SA: Strengthen In-context Learning via Submodular Selective Annotationcited by 4
- 2025CBIL: Collective Behavior Imitation Learning for Fish from Real Videoscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Increasing the Thinking Budget is Not All You Need
- 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents
- 2025Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templates
- 2026The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits
- 2025Chain-of-Thought Tokens are Computer Program Variables
- 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
- 2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
- 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning