mlsanityData to 7 Oct 2026

ROSE: An Intent-Centered Evaluation Metric for NL2SQL

arXiv 2604.12988 ↗First posted 14 Apr 2026cs.DB cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 90 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

22 of 90 references.

References · 22

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  3. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  4. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  5. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  6. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
  7. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  8. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  9. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  10. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 51
  11. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 51
  12. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 49
10 more
  1. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 32
  2. 2025NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translationcited by 31
  3. 2025DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Frameworkcited by 26
  4. 2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 25
  5. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  6. 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
  7. 2024VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraintscited by 8
  8. 2020TADOC: Text Analytics Directly on Compressioncited by 3
  9. 2025Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Modelscited by 2
  10. 2026NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutionscited by 1

Built on by

Not cited yet.