# Evaluating Cross-Domain Text-to-SQL Models and Benchmarks

arXiv: https://arxiv.org/abs/2310.18538 · HTML: https://mlsanity.com/paper/2310.18538/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2012.12627/) (2020)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites](https://mlsanity.com/paper/2010.02840/) (2020)
- [LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations](https://mlsanity.com/paper/2106.01093/) (2021)
- [A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://mlsanity.com/paper/2303.13547/) (2023)
- [Towards Robustness of Text-to-SQL Models against Synonym Substitution](https://mlsanity.com/paper/2106.01065/) (2021)
- [Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing](https://mlsanity.com/paper/2301.07507/) (2023)
- [Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization](https://mlsanity.com/paper/2109.05157/) (2021)
- [Grounded Adaptation for Zero-shot Executable Semantic Parsing](https://mlsanity.com/paper/2009.07396/) (2020)
- [SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task](https://mlsanity.com/paper/1810.05237/) (2018)

## Built on by (most cited first, 30 of 32)

- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation](https://mlsanity.com/paper/2405.07467/) (2024)
- [Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://mlsanity.com/paper/2507.02825/) (2025)
- [AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries](https://mlsanity.com/paper/2406.19073/) (2024)
- [Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark](https://mlsanity.com/paper/2402.12243/) (2024)
- [SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models](https://mlsanity.com/paper/2403.02951/) (2024)
- [SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging](https://mlsanity.com/paper/2408.12733/) (2024)
- [Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing](https://mlsanity.com/paper/2502.18448/) (2025)
- [Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards](https://mlsanity.com/paper/2601.08778/) (2026)
- [I Need Help! Evaluating LLM's Ability to Ask for Users' Support: A Case Study on Text-to-SQL Generation](https://mlsanity.com/paper/2407.14767/) (2024)
- [FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark](https://mlsanity.com/paper/2409.19014/) (2024)
- [AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL](https://mlsanity.com/paper/2508.15276/) (2025)
- [Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges](https://mlsanity.com/paper/2508.00217/) (2025)
- [Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2603.16140/) (2026)
- [FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents](https://mlsanity.com/paper/2605.02815/) (2026)
- [M3: Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis](https://mlsanity.com/paper/2507.01053/) (2025)
- [ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities](https://mlsanity.com/paper/2603.29399/) (2026)
- [Rationalization Models for Text-to-SQL](https://mlsanity.com/paper/2502.06759/) (2025)
- [Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity](https://mlsanity.com/paper/2601.15728/) (2026)
- [Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering](https://mlsanity.com/paper/2603.20004/) (2026)
- [NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions](https://mlsanity.com/paper/2604.16493/) (2026)
- [Beyond Text-to-SQL: An Agentic LLM System for Governed Enterprise Analytics APIs](https://mlsanity.com/paper/2605.21027/) (2026)
- [SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation](https://mlsanity.com/paper/2609.08950/) (2026)
- [The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators](https://mlsanity.com/paper/2609.21133/) (2026)
- [SPORTSQL: An Interactive System for Real-Time Sports Reasoning and Visualization](https://mlsanity.com/paper/2508.17157/) (2025)
- [DynaQuery: A Self-Adapting Framework for Querying Structured and Multimodal Data](https://mlsanity.com/paper/2510.18029/) (2025)
- [RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation](https://mlsanity.com/paper/2602.07086/) (2026)
- [Dial: A Knowledge-Grounded Dialect-Specific NL2SQL System](https://mlsanity.com/paper/2603.07449/) (2026)
- [Anatomy of a Query: W5H Dimensions and FAR Patterns for Text-to-SQL Evaluation](https://mlsanity.com/paper/2605.05525/) (2026)
- [Residual Skill Optimization for Text-to-SQL Ensembles](https://mlsanity.com/paper/2605.21792/) (2026)

## Abstract (arXiv)

Text-to-SQL benchmarks play a crucial role in evaluating the progress made in the field and the ranking of different models. However, accurately matching a model-generated SQL query to a reference SQL query in a benchmark fails for various reasons, such as underspecified natural language queries, inherent assumptions in both model-generated and reference queries, and the non-deterministic nature of SQL output under certain conditions. In this paper, we conduct an extensive study of several prominent cross-domain text-to-SQL benchmarks and re-evaluate some of the top-performing models within these benchmarks, by both manually evaluating the SQL queries and rewriting them in equivalent expressions. Our evaluation reveals that attaining a perfect performance on these benchmarks is unfeasible due to the multiple interpretations that can be derived from the provided samples. Furthermore, we find that the true performance of the models is underestimated and their relative performance changes after a re-evaluation. Most notably, our evaluation reveals a surprising discovery: a recent GPT4-based model surpasses the gold standard reference queries in the Spider benchmark in our human evaluation. This finding highlights the importance of interpreting benchmark evaluations cautiously, while also acknowledging the critical role of additional independent evaluations in driving advancements in the field.