mlsanityData to 14 Oct 2026

Anatomy of a Query: W5H Dimensions and FAR Patterns for Text-to-SQL Evaluation

arXiv 2605.05525 ↗First posted 7 May 2026cs.DB cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural language interfaces to databases have gained popularity, yet the theoretical foundations for evaluating and designing these systems remain underdeveloped. We present QUEST (Query Understanding Evaluation through Semantic Translation), a framework resting on two independently motivated components: the FAR structural invariant, which holds that every well-formed query reduces to Filter, Aggregate, and Return operations; and the W5H dimensional framework, which holds that all filtering criteria map to six semantic dimensions (Who, What, Where, When, Why, and How). Validated across five text-to-SQL datasets (n = 120,464), FAR conformance is universal across all domains and schema types, while W5H dimensional profiles vary substantially. Healthcare queries are strongly concentrated in temporal (WHEN: 80.4%) and person-centric (WHO: 73.0%) dimensions far exceeding general-domain benchmarks, and causal (WHY) and mechanistic (HOW) reasoning are near-zero everywhere, with apparent HOW exceptions reflecting quantitative aggregation rather than genuine procedural reasoning. These results identify a frontier that must be crossed for genuine machine reasoning over structured data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 19 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

8 of 19 references.

References · 8

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  5. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  6. 2023EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Recordscited by 45
  7. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  8. 2013Lambda Dependency-Based Compositional Semanticscited by 26

Built on by

Not cited yet.