# SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification

arXiv: https://arxiv.org/abs/2510.26840 · HTML: https://mlsanity.com/paper/2510.26840/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites](https://mlsanity.com/paper/2010.02840/) (2020)
- [A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://mlsanity.com/paper/2303.13547/) (2023)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings](https://mlsanity.com/paper/2305.11853/) (2023)
- [RSL-SQL: Robust Schema Linking in Text-to-SQL Generation](https://mlsanity.com/paper/2411.00073/) (2024)
- [A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL](https://mlsanity.com/paper/2411.08599/) (2024)
- [Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark](https://mlsanity.com/paper/2402.12243/) (2024)
- [Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search](https://mlsanity.com/paper/2502.17248/) (2025)
- [Automatic Metadata Extraction for Text-to-SQL](https://mlsanity.com/paper/2505.19988/) (2025)
- [A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases](https://mlsanity.com/paper/2311.07509/) (2023)
- [CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning](https://mlsanity.com/paper/2505.13271/) (2025)
- [Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuning](https://mlsanity.com/paper/2505.14174/) (2025)
- [Axiomatic Foundations and Algorithms for Deciding Semantic Equivalences of SQL Queries](https://mlsanity.com/paper/1802.02229/) (2018)
- [HoTTSQL: Proving Query Rewrites with Univalent SQL Semantics](https://mlsanity.com/paper/1607.04822/) (2016)
- [SLM-SQL: An Exploration of Small Language Models for Text-to-SQL](https://mlsanity.com/paper/2507.22478/) (2025)
- [VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints](https://mlsanity.com/paper/2403.03193/) (2024)
- [Data Generation for Testing and Grading SQL Queries](https://mlsanity.com/paper/1411.6704/) (2014)
- [Verifying SQL Queries using Theories of Tables and Relations](https://mlsanity.com/paper/2405.03057/) (2024)
- [Data Generation for Testing Complex Queries](https://mlsanity.com/paper/2409.18821/) (2024)

## Built on by (most cited first, 4 of 4)

- [FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents](https://mlsanity.com/paper/2605.02815/) (2026)
- [SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints](https://mlsanity.com/paper/2603.04334/) (2026)
- [Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL](https://mlsanity.com/paper/2609.25938/) (2026)
- [VeriPIE: Checkable Query Plans and Evidence-Guided Selection for Text-to-SQL](https://mlsanity.com/paper/2610.17493/) (2026)

## Abstract (arXiv)

Community-driven Text-to-SQL evaluation platforms play a pivotal role in tracking the state of the art of Text-to-SQL performance. The reliability of the evaluation process is critical for driving progress in the field. Current evaluation methods are largely test-based, which involves comparing the execution results of a generated SQL query and a human-labeled ground-truth on a static test database. Such an evaluation is optimistic, as two queries can coincidentally produce the same output on the test database while actually being different. In this work, we propose a new alternative evaluation pipeline, called SpotIt, where a formal bounded equivalence verification engine actively searches for a database that differentiates the generated and ground-truth SQL queries. We develop techniques to extend existing verifiers to support a richer SQL subset relevant to Text-to-SQL. A performance evaluation of ten Text-to-SQL methods on the high-profile BIRD dataset suggests that test-based methods can often overlook differences between the generated query and the ground-truth. Further analysis of the verification results reveals a more complex picture of the current Text-to-SQL evaluation.