mlsanityData to 7 Oct 2026

STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases

arXiv 2509.19508 ↗First posted 23 Sep 2025cs.DB cs.CLcited by 2

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Semantic parsing methods for converting text to SQL queries enable question answering over structured data and can greatly benefit analysts who routinely perform complex analytics on vast data stored in specialized relational databases. Although several benchmarks measure the abilities of text to SQL, the complexity of their questions is inherently limited by the level of expressiveness in query languages and none focus explicitly on questions involving complex analytical reasoning which require operations such as calculations over aggregate analytics, time series analysis or scenario understanding. In this paper, we introduce STARQA, the first public human-created dataset of complex analytical reasoning questions and answers on three specialized-domain databases. In addition to generating SQL directly using LLMs, we evaluate a novel approach (Text2SQLCode) that decomposes the task into a combination of SQL and Python: SQL is responsible for data fetching, and Python more naturally performs reasoning. Our results demonstrate that identifying and combining the abilities of SQL and Python is beneficial compared to using SQL alone, yet the dataset still remains quite challenging for the existing state-of-the-art LLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 7 of its 35 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

7 of 35 references.

References · 7

  1. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  2. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  3. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  4. 2023ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systemscited by 42
  5. 2024End-to-end Text-to-SQL Generation within an Analytics Insight Enginecited by 7
  6. 2025Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequenciescited by 5
  7. 2025Balancing Content Size in RAG-Text2SQL Systemcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GrepSeek: Training Search Agents for Direct Corpus Interactioncited by 10
  2. 2026CAPER: Clause-Aligned Process Supervision for Text-to-SQLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Weaver: Interweaving SQL and LLM for Table Reasoning
  2. 2025LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning
  3. 2022A Benchmark for Generalizable and Interpretable Temporal Question Answering over Knowledge Bases
  4. 2025CRED-SQL: Enhancing Real-world Large Scale Database Text-to-SQL Parsing through Cluster Retrieval and Execution Description
  5. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
  6. 2024Multi-Turn Interactions for Text-to-SQL with Large Language Models
  7. 2026Integrating Reasoning and Generalization in Text-to-SQL via Self-Enhanced Fine-Tuning
  8. 2024TANQ: An open domain dataset of table answered questions