Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number. No execution-match metric can score them. This paper introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed. QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour, and gates every answer on deterministic post-execution checks. On an 80-task synthetic test split evaluated once, QueryProof outperforms a direct-prompted 32B baseline by +0.237 [+0.112, +0.375] Business Truth Rate at 71.0% lower cost per correct answer; against a cost-matched few-shot baseline the accuracy gain holds but the cost difference does not resolve. This compares systems rather than model sizes: the 32B baseline receives none of the scaffolding. False success falls from 0.754 to 0.351 of returned answers, and no wrong number was returned on an answerable task (0 of 24), though 13 answers went to questions requiring clarification or abstention. Removing the routing layer changes little (0.562 against 0.537), so the result does not depend on escalation. Routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced. Resampling template families rather than tasks widens both accuracy intervals to include zero, so the effect's direction is better supported than its magnitude. The gain tracks the deterministic layer, though no component ablation was run.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 27 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
12 of 27 references.
References · 12
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
- 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
- 2018Robust Text-to-SQL Generation with Execution-Guided Decodingcited by 51
- 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 31
- 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 15
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence
- 2025Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5
- 2026Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
- 2024Tool-Assisted Agent on SQL Inspection and Refinement in Real-World Scenarios
- 2026ACTS-SQL: Agentic and Critic-Oriented Tree-Structured SQL Correctness with Large Language Models
- 2026AgentNLQ: A General-Purpose Agent for Natural Language to SQL
- 2026A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark
- 2026Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact