mlsanityData to 7 Oct 2026

FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark

arXiv 2409.19014 ↗First posted 24 Sep 2024cs.CL cs.IR cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Text-to-SQL systems have become crucial for translating natural language into SQL queries in various industries, enabling non-technical users to perform complex data operations. The need for accurate evaluation methods has increased as these systems have grown more sophisticated. However, the Execution Accuracy (EX), the most prevalent evaluation metric, still shows many false positives and negatives. Thus, this paper introduces FLEX (False-Less EXecution), a novel approach to evaluating text-to-SQL systems using large language models (LLMs) to emulate human expert-level evaluation of SQL queries. Our metric improves agreement with human experts (from 62 to 87.04 in Cohen's kappa) with comprehensive context and sophisticated criteria. Our extensive experiments yield several key insights: (1) Models' performance increases by over 2.6 points on average, substantially affecting rankings on Spider and BIRD benchmarks; (2) The underestimation of models in EX primarily stems from annotation quality issues; and (3) Model performance on particularly challenging questions tends to be overestimated. This work contributes to a more accurate and nuanced evaluation of text-to-SQL systems, potentially reshaping our understanding of state-of-the-art performance in this field.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

31 of 49 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
19 more
  1. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  5. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  6. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  7. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  8. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  9. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  10. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
  11. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  12. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  13. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  14. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  15. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 49
  16. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 33
  17. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  18. 2024Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translationcited by 21
  19. 2023LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregationcited by 3
  2. 2025Access Paths for Efficient Ordering with Large Language Modelscited by 1
  3. 2026Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguitycited by 1
  4. 2026Fine-Grained Table Retrieval Through the Lens of Complex Queriescited by 1
  5. 2026SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generationcited by 1
  6. 2026ROSE: An Intent-Centered Evaluation Metric for NL2SQLcited by 0
  7. 2026Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systemscited by 0
  8. 2026BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoningcited by 0
  9. 2026Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQLcited by 0
  10. 2026Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQLcited by 0
  11. 2026Closing the Cross-Dialect Gap: Query Plans as a Portable Interface in Text-to-SQLcited by 0