# SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints

arXiv: https://arxiv.org/abs/2603.04334 · HTML: https://mlsanity.com/paper/2603.04334/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [RSL-SQL: Robust Schema Linking in Text-to-SQL Generation](https://mlsanity.com/paper/2411.00073/) (2024)
- [Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search](https://mlsanity.com/paper/2502.17248/) (2025)
- [CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning](https://mlsanity.com/paper/2505.13271/) (2025)
- [Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuning](https://mlsanity.com/paper/2505.14174/) (2025)
- [Axiomatic Foundations and Algorithms for Deciding Semantic Equivalences of SQL Queries](https://mlsanity.com/paper/1802.02229/) (2018)
- [HoTTSQL: Proving Query Rewrites with Univalent SQL Semantics](https://mlsanity.com/paper/1607.04822/) (2016)
- [SLM-SQL: An Exploration of Small Language Models for Text-to-SQL](https://mlsanity.com/paper/2507.22478/) (2025)
- [VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints](https://mlsanity.com/paper/2403.03193/) (2024)
- [SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification](https://mlsanity.com/paper/2510.26840/) (2025)

## Built on by (most cited first, 1 of 1)

- [Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL](https://mlsanity.com/paper/2609.25938/) (2026)

## Abstract (arXiv)

We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a best-effort constraint-mining pipeline that combines rule-based specification mining with LLM-based validation over example databases. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.