# ROSE: An Intent-Centered Evaluation Metric for NL2SQL

arXiv: https://arxiv.org/abs/2604.12988 · HTML: https://mlsanity.com/paper/2604.12988/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 90 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 22 of 22)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [CodeS: Towards Building Open-source Language Models for Text-to-SQL](https://mlsanity.com/paper/2402.16347/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [C3: Zero-shot Text-to-SQL with ChatGPT](https://mlsanity.com/paper/2307.07306/) (2023)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites](https://mlsanity.com/paper/2010.02840/) (2020)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment](https://mlsanity.com/paper/2502.14913/) (2025)
- [RSL-SQL: Robust Schema Linking in Text-to-SQL Generation](https://mlsanity.com/paper/2411.00073/) (2024)
- [Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation](https://mlsanity.com/paper/2405.15307/) (2024)
- [AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries](https://mlsanity.com/paper/2406.19073/) (2024)
- [NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation](https://mlsanity.com/paper/2503.11984/) (2025)
- [DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework](https://mlsanity.com/paper/2510.17586/) (2025)
- [Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewards](https://mlsanity.com/paper/2505.04671/) (2025)
- [EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing](https://mlsanity.com/paper/2503.22402/) (2025)
- [FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark](https://mlsanity.com/paper/2409.19014/) (2024)
- [VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints](https://mlsanity.com/paper/2403.03193/) (2024)
- [TADOC: Text Analytics Directly on Compression](https://mlsanity.com/paper/2009.09442/) (2020)
- [Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Models](https://mlsanity.com/paper/2503.17811/) (2025)
- [NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions](https://mlsanity.com/paper/2604.16493/) (2026)

## Abstract (arXiv)

Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.