Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance
On the map: LLM agents and tool use → Tool learning and function calling
Abstract (arXiv)
Tool use has become central to modern LLM agents, yet interface design is rarely isolated as an experimental variable. This paper studies whether schema based tool contracts and structured validation diagnostics improve reliability under strict interaction budgets. We evaluate three conditions that preserve identical tool semantics and information content: free form documentation, JSON Schema specifications, and JSON Schema with structured diagnostics. We implement a deterministic software engineering sandbox with logs, metrics, configurations, and repository tasks, and evaluate a fully crossed pilot with one open local model, three seeds, three interface conditions, and four budgets. We report end task success, interface misuse, execution failures, semantic misuse, recovery behavior, and overhead. In this pilot, success remains zero across conditions, while schema conditions reduce interface misuse but not semantic misuse. The evidence supports a precise interpretation that interface formalization improves contract adherence, but semantic action quality and timeout sensitive tasks remain dominant bottlenecks under constrained local inference.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 12 of its 15 references that are in the map. First posted on arXiv 12 Mar 2026.
Builds on
12 of 15 references.
Closest ancestors · 3
Same direction, newest first
- 2023Gorilla: Large Language Model Connected with Massive APIs24 May
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
- 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb
Other references · 9
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2023Augmented Language Models: a Surveycited by 403
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injectioncited by 5
- 2026ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systemscited by 4
- 2026Guardrails as Infrastructure: Policy-First Control for Tool-Orchestrated Workflowscited by 1
- 2026Callability Is Not Operability: Controlled Interface Interventions for LLM Agentscited by 1
- 2026What Stops a Small Language Model From Driving a Database Agentcited by 0
- 2026A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolutioncited by 0
- 2026Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Executioncited by 0
- 2026Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution13 Oct 2026
- 2026A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution11 Oct 2026
- 2026What Stops a Small Language Model From Driving a Database Agent18 Sep 2026
- 2026Callability Is Not Operability: Controlled Interface Interventions for LLM Agents23 Aug 2026
- 2026AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection7 Aug 2026
- 2026ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems2 Apr 2026
- 2026Guardrails as Infrastructure: Policy-First Control for Tool-Orchestrated Workflows18 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026ContractBench: Can LLM Agents Preserve Observation Contracts?
- 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
- 2026TSCG: Deterministic Tool-Schema Compilation for Agentic LLM Deployments
- 2026When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents
- 2026Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
- 2026How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines
- 2026Action-Space Shaping for LLM Agents: Measuring and Mitigating Tool-Schema Bias
- 2026MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents