# The Bitter Lesson of Tool Calling

arXiv: https://arxiv.org/abs/2608.06370 · HTML: https://mlsanity.com/paper/2608.06370/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 8 of 8)

- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://mlsanity.com/paper/2401.17167/) (2024)
- [API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs](https://mlsanity.com/paper/2402.15491/) (2024)
- [CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions](https://mlsanity.com/paper/2506.01859/) (2025)
- [On Effectiveness and Efficiency of Agentic Tool-calling and RL Training](https://mlsanity.com/paper/2606.00135/) (2026)
- [The Deterministic Horizon: When Extended Reasoning Fails and Tool Delegation Becomes Necessary](https://mlsanity.com/paper/2606.00376/) (2026)
- [HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents](https://mlsanity.com/paper/2606.13663/) (2026)

## Built on by (most cited first, 2 of 2)

- [Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents](https://mlsanity.com/paper/2609.11999/) (2026)
- [ProgAgent: Learning Tool Orchestration through Programmatic Training](https://mlsanity.com/paper/2610.14774/) (2026)

## Abstract (arXiv)

Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been conducted. In this work, we empirically compare programmatic tool calling (PTC) to native JSON tool calling across 14 language models on BFCL v4. In the programmatic tool calling paradigm, tools are exposed as typed Python stubs that the model invokes through code, with execution and results handled in a single agent turn. Programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models on BFCL v4, with the GPT-5.6 family achieving a 10.6% improvement over the JSON tool calling baseline. Further, it matches or outperforms baseline in 13 of 14 models under parallel fan-out, and holds stable under context rot conditions where baseline degrades 2.3% on average. Our results demonstrate that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.