# STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases

arXiv: https://arxiv.org/abs/2509.19508 · HTML: https://mlsanity.com/paper/2509.19508/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 7 of 7)

- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage](https://mlsanity.com/paper/2308.03427/) (2023)
- [ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems](https://mlsanity.com/paper/2306.04743/) (2023)
- [End-to-end Text-to-SQL Generation within an Analytics Insight Engine](https://mlsanity.com/paper/2406.12104/) (2024)
- [Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequencies](https://mlsanity.com/paper/2505.23804/) (2025)
- [Balancing Content Size in RAG-Text2SQL System](https://mlsanity.com/paper/2502.15723/) (2025)

## Built on by (most cited first, 2 of 2)

- [GrepSeek: Training Search Agents for Direct Corpus Interaction](https://mlsanity.com/paper/2605.29307/) (2026)
- [CAPER: Clause-Aligned Process Supervision for Text-to-SQL](https://mlsanity.com/paper/2606.03327/) (2026)

## Abstract (arXiv)

Semantic parsing methods for converting text to SQL queries enable question answering over structured data and can greatly benefit analysts who routinely perform complex analytics on vast data stored in specialized relational databases. Although several benchmarks measure the abilities of text to SQL, the complexity of their questions is inherently limited by the level of expressiveness in query languages and none focus explicitly on questions involving complex analytical reasoning which require operations such as calculations over aggregate analytics, time series analysis or scenario understanding. In this paper, we introduce STARQA, the first public human-created dataset of complex analytical reasoning questions and answers on three specialized-domain databases. In addition to generating SQL directly using LLMs, we evaluate a novel approach (Text2SQLCode) that decomposes the task into a combination of SQL and Python: SQL is responsible for data fetching, and Python more naturally performs reasoning. Our results demonstrate that identifying and combining the abilities of SQL and Python is beneficial compared to using SQL alone, yet the dataset still remains quite challenging for the existing state-of-the-art LLMs.