mlsanityData to 7 Oct 2026

Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models

arXiv 2604.25149 ↗First posted 28 Apr 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures - incorrect answers and confident hallucinations - both rooted in the same cause: the model is forced to infer business semantics that the schema does not encode. We test whether supplying those semantics as context closes the gap. We benchmark three frontier LLMs (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4) on 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse, using a paired single-shot protocol. Each model is evaluated twice: once given only the warehouse schema, and once given the schema plus a 4 KB hand-authored markdown document describing the dataset's measures, conventions, and disambiguation rules. Adding the document improves accuracy by +17 to +23 percentage points across all three models. With it, the three models are statistically indistinguishable (67.7-68.7%); without it, they are also indistinguishable (45.5-50.5%). Every cross-cluster comparison is significant at p < 0.01. The presence of the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not. We interpret this as a structural result: explicit business semantics suppress the dominant class of text-to-SQL errors not by making the model more capable, but by changing what the model is being asked to do.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

38 of 45 references.

References · 38

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  4. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  5. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  6. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  7. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  8. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  9. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  10. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  11. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  12. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
26 more
  1. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  2. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  4. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  5. 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
  6. 2024SynCode: LLM Generation with Grammar Augmentationcited by 66
  7. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  8. 2024Grammar-Aligned Decodingcited by 54
  9. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  10. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 49
  11. 2024BEAVER: An Enterprise Benchmark for Text-to-SQLcited by 37
  12. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 33
  13. 2025NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translationcited by 31
  14. 2024IterGen: Iterative Semantic-aware Structured LLM Generation with Backtrackingcited by 28
  15. 2023A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databasescited by 25
  16. 2025Automatic Metadata Extraction for Text-to-SQLcited by 25
  17. 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 19
  18. 2024Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!cited by 18
  19. 2025Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errorscited by 14
  20. 2023Error Detection for Text-to-SQL Semantic Parsingcited by 11
  21. 2025Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testingcited by 8
  22. 2024Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Datacited by 7
  23. 2025LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoningcited by 7
  24. 2025SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generationcited by 4
  25. 2024Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQLcited by 2
  26. 2024Beyond Fine-Tuning: Effective Strategies for Mitigating Hallucinations in Large Language Models for Data Analyticscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQLcited by 0
  2. 2026Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agentscited by 0
  3. 2026Enterprise Representation Simplification (ERS): Reducing Representational Complexity for Enterprise AIcited by 0