Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows

arXiv 2607.00828 ↗First posted 1 Jul 2026cs.DB cs.AIcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 11 of its 30 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

11 of 30 references.

References · 11

2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
2025XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQLcited by 40
2025Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AIcited by 9
2025GenEdit: Compounding Operators and Continuous Improvement to Tackle Text-to-SQL in the Enterprisecited by 4
2025Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysiscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way