# Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning

arXiv: https://arxiv.org/abs/2402.12554 · HTML: https://mlsanity.com/paper/2402.12554/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2012.12627/) (2020)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Learning a Neural Semantic Parser from User Feedback](https://mlsanity.com/paper/1704.08760/) (2017)
- [Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing](https://mlsanity.com/paper/2301.07507/) (2023)
- [Knowledge-aware Zero-Shot Learning: Survey and Perspective](https://mlsanity.com/paper/2103.00070/) (2021)
- [Zero-shot Visual Question Answering using Knowledge Graph](https://mlsanity.com/paper/2107.05348/) (2021)
- [Commonsense Properties from Query Logs and Question Answering Forums](https://mlsanity.com/paper/1905.10989/) (2019)
- [Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledge](https://mlsanity.com/paper/2301.01067/) (2023)
- [Knowledge-based Transfer Learning Explanation](https://mlsanity.com/paper/1807.08372/) (2018)
- [BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering](https://mlsanity.com/paper/2305.15932/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?](https://mlsanity.com/paper/2506.21215/) (2025)
- [Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts](https://mlsanity.com/paper/2405.06524/) (2024)
- [MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.05587/) (2025)
- [Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languages](https://mlsanity.com/paper/2506.00912/) (2025)
- [Transduction is All You Need for Structured Data Workflows](https://mlsanity.com/paper/2508.15610/) (2025)
- [OraPlan-SQL: A Planning-Centric Framework for Complex Bilingual NL2SQL Reasoning](https://mlsanity.com/paper/2510.23870/) (2025)
- [PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism](https://mlsanity.com/paper/2605.07796/) (2026)
- [Dial: A Knowledge-Grounded Dialect-Specific NL2SQL System](https://mlsanity.com/paper/2603.07449/) (2026)
- [Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain](https://mlsanity.com/paper/2603.17815/) (2026)
- [SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing](https://mlsanity.com/paper/2606.11424/) (2026)
- [SDAM: Structure-Difference-Aware Memory Evolution for Complex Text-to-SQL](https://mlsanity.com/paper/2608.12338/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)

## Abstract (arXiv)

We present Archer, a challenging bilingual text-to-SQL dataset specific to complex reasoning, including arithmetic, commonsense and hypothetical reasoning. It contains 1,042 English questions and 1,042 Chinese questions, along with 521 unique SQL queries, covering 20 English databases across 20 domains. Notably, this dataset demonstrates a significantly higher level of complexity compared to existing publicly available datasets. Our evaluation shows that Archer challenges the capabilities of current state-of-the-art models, with a high-ranked model on the Spider leaderboard achieving only 6.73% execution accuracy on Archer test set. Thus, Archer presents a significant challenge for future research in this field.