# UNITE: A Unified Benchmark for Text-to-SQL Evaluation

arXiv: https://arxiv.org/abs/2305.16265 · HTML: https://mlsanity.com/paper/2305.16265/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 21 of 21)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [SParC: Cross-Domain Semantic Parsing in Context](https://mlsanity.com/paper/1906.02285/) (2019)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)
- [Towards Robustness of Text-to-SQL Models against Synonym Substitution](https://mlsanity.com/paper/2106.01065/) (2021)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Evaluating the Text-to-SQL Capabilities of Large Language Models](https://mlsanity.com/paper/2204.00498/) (2022)
- [SmBoP: Semi-autoregressive Bottom-up Semantic Parsing](https://mlsanity.com/paper/2010.12412/) (2020)
- [Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization](https://mlsanity.com/paper/2109.05157/) (2021)
- [RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL](https://mlsanity.com/paper/2205.06983/) (2022)
- [On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries](https://mlsanity.com/paper/2010.11246/) (2020)
- [SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising](https://mlsanity.com/paper/2105.07911/) (2021)
- [An End-to-end Neural Natural Language Interface for Databases](https://mlsanity.com/paper/1804.00401/) (2018)
- [CL Scholar: The ACL Anthology Knowledge Graph Miner](https://mlsanity.com/paper/1804.05514/) (2018)

## Built on by (most cited first, 14 of 14)

- [BEAVER: An Enterprise Benchmark for Text-to-SQL](https://mlsanity.com/paper/2409.02038/) (2024)
- [Selective Demonstrations for Cross-domain Text-to-SQL](https://mlsanity.com/paper/2310.06302/) (2023)
- [Semantic Decomposition of Question and SQL for Text-to-SQL Parsing](https://mlsanity.com/paper/2310.13575/) (2023)
- [A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challenges](https://mlsanity.com/paper/2412.05208/) (2024)
- [Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Prompting](https://mlsanity.com/paper/2308.02582/) (2023)
- [LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning](https://mlsanity.com/paper/2505.18744/) (2025)
- [Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT -- A Text-to-SQL Parsing Comparison](https://mlsanity.com/paper/2310.10190/) (2023)
- [SQaLe: A Large Text-to-SQL Corpus Grounded in Real Schemas](https://mlsanity.com/paper/2602.22223/) (2025)
- [A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability](https://mlsanity.com/paper/2502.02866/) (2025)
- [SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps](https://mlsanity.com/paper/2510.27532/) (2025)
- [LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL](https://mlsanity.com/paper/2510.02350/) (2025)
- [Retrieval and Augmentation of Domain Knowledge for Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2510.02394/) (2025)
- [A Multi-agent Text2SQL Framework using Small Language Models and Execution Feedback](https://mlsanity.com/paper/2512.18622/) (2025)
- [BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning](https://mlsanity.com/paper/2606.02109/) (2026)

## Abstract (arXiv)

A practical text-to-SQL system should generalize well on a wide variety of natural language questions, unseen database schemas, and novel SQL query structures. To comprehensively evaluate text-to-SQL systems, we introduce a UNIfied benchmark for Text-to-SQL Evaluation (UNITE). It is composed of publicly available text-to-SQL datasets, containing natural language questions from more than 12 domains, SQL queries from more than 3.9K patterns, and 29K databases. Compared to the widely used Spider benchmark, we introduce ~120K additional examples and a threefold increase in SQL patterns, such as comparative and boolean questions. We conduct a systematic study of six state-of-the-art (SOTA) text-to-SQL parsers on our new benchmark and show that: 1) Codex performs surprisingly well on out-of-domain datasets; 2) specially designed decoding methods (e.g. constrained beam search) can improve performance for both in-domain and out-of-domain settings; 3) explicitly modeling the relationship between questions and schemas further improves the Seq2Seq models. More importantly, our benchmark presents key challenges towards compositional generalization and robustness issues -- which these SOTA models cannot address well. Our code and data processing script are available at https://github.com/awslabs/unified-text2sql-benchmark