The Remarkable Effectiveness of Providing AI Agents with Natural Language Tools: A Replication Study Validating NLT Performance Across 14 Models
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
This study independently replicates and extends the Natural Language Tools (NLT) framework of Johnson et al.~(2025), which questions the use of structured tool calling in large language model (LLM) agentic systems. We evaluated NLT across 14 models and 8,560 trials, adding newer frontier, reasoning, and open-weight models to the original set. The results confirm the core findings and add detail. NLT improves tool-calling accuracy by 14.9 percentage points overall (62.3% versus 47.4% structured) and reduces critical errors by 93% (51 versus 755 errors). The gains depend on model capability: models without native tool calling, reasoning models, and smaller models gain substantially (+24.0pp to +43.1pp), while heavily optimized frontier models (GPT-5, Gemini 2.5 Pro) show smaller or reversed advantages. This matches recent analyses of reinforcement-learning-optimized tool use (Martinez, 2025). NLT also cuts token usage by 25.2%. The reliability and efficiency advantages compound in recursive agentic workflows, where agents chain many tool calls across sub-agents: a structured failure triggers retries, fallback routing, and coordination overhead, while NLT avoids most of that cost at the source. This work makes three contributions: (1) the first independent validation of NLT using open-source tooling, (2) evidence that model capability moderates NLT's advantages (Chen et al., 2025; Zhang et al., 2025), and (3) a measurement of NLT's reliability benefit (93% fewer errors), its most deployment-relevant property given the known fragility of structured tool calling. NLT is a practical alternative to structured tool calling, especially for production systems that value reliability over parseability.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 11 of its 23 references that are in the map. First posted on arXiv 4 Jul 2026.
Builds on
11 of 23 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
- 2023Augmented Language Models: a Surveycited by 403
- 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
- 2023Tool Learning with Foundation Modelscited by 302
- 2023System 2 Attention (is something you might need too)cited by 89
- 2024ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesiscited by 30
- 2025CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routingcited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents
- 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
- 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
- 2026Agentic Tool Use in Large Language Models
- 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
- 2025Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning
- 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents