What Stops a Small Language Model From Driving a Database Agent
On the map: LLM agents and tool use → Tool learning and function calling also fits: General LLM agent frameworks and benchmarks
Abstract (arXiv)
Small open-weight language models are assumed to fail at agentic database work because they lack the reasoning capacity for it. We test that against a production system. Over eleven days we drove the agent mode of an open-source SQL client with 39 open-weight models served locally and one hosted control, across six task surfaces: 8,199 runs, 110,711 ledger events, 14,008 refused tool calls. Of the 2,100 model-attributed agent-mode losses, 1,590, or 75.7%, came from runs that had invoked at least one tool. That majority is what survives resampling models rather than runs: it holds in 99.7% of clustered resamples and in 15 of the 22 models with at least twenty losses. Within it, transport, a run that used the tools and never got a deliverable through, is the largest class at 36.2% and capability, a run that invoked no tool at all, the smallest at 17.3%; we report that ordering as a property of this corpus rather than a general finding, since clustered by model it holds in only 74.5% of resamples. Transport failures decompose into a few mechanical argument shapes. Production ledgers record refusal codes and never the model's arguments, so these were invisible for ten days; capturing them exposed five server defects, one of which demanded a field on one tool, forbade it on the sibling that composed it, then failed the run for its absence. Five server changes, touching no model, prompt or sampling setting, moved six models by 6 to 21 cells out of 30. We also report a confound we believe affects published local-model benchmarks, ours included: with no context cap, one 7.1 GB model was admitted at its full 262,144-token window and held 51 GB on a 64 GB machine, producing runs indistinguishable in any ordinary log from a model timing out. The corpus, the scorer and a verifier that regenerates every figure are released.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 26 of its 29 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
26 of 29 references.
Closest ancestors · 10
Same direction, newest first
- 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail24 Aug
- 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents6 Jul
- 2026ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems2 Apr
- 2026Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance12 Mar
- 2026Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errors22 Jan
- 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases25 Sep
- 2025Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems21 Jul
- 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
Other references · 16
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
- 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
- 2025Small Language Models are the Future of Agentic AIcited by 270
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
- 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
- 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
- 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
4 more
- 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
- 2023D-Bot: Database Diagnosis System using Large Language Modelscited by 28
- 2026AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCPcited by 5
- 2025AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systemscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
- 2026Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows
- 2026Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms
- 2026Harnessing LLMs as Agents: What Does It Cost?
- 2025Small Language Models for Agentic Systems: A Survey of Architectures, Capabilities, and Deployment Trade offs
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
- 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents
- 2026Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?