mlsanityData to 7 Oct 2026

Semantic Evaluation for Text-to-SQL with Distilled Test Suites

arXiv 2010.02840 ↗First posted 6 Oct 2020cs.CL cs.AIcited by 123

Not on the map. Why?

Abstract (arXiv)

We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models. Our method distills a small test suite of databases that achieves high code coverage for the gold query from a large number of randomly generated databases. At evaluation time, it computes the denotation accuracy of the predicted queries on the distilled test suite, hence calculating a tight upper-bound for semantic accuracy efficiently. We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples. In contrast, the current Spider metric leads to a 2.5% false negative rate on average and 8.1% in the worst case, indicating that test suite accuracy is needed. Our implementation, along with distilled test suites for eleven Text-to-SQL datasets, is publicly available.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 28; 2024: 32; 2025: 23; 2026: 20.

123 papers.

Sort:

By area:

  1. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  2. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  3. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  5. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  6. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  7. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  8. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  9. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  10. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  11. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 158
  12. 2022Natural Language to Code Translation with Executioncited by 122
38 more
  1. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 118
  2. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  3. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  4. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  5. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  6. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  7. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 53
  8. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  9. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 51
  10. 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
  11. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  12. 2023SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)cited by 42
  13. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  14. 2024FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysiscited by 35
  15. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  16. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  17. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 32
  18. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  19. 2024Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigmcited by 31
  20. 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
  21. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  22. 2024PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistencycited by 26
  23. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  24. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  25. 2024Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translationcited by 21
  26. 2023Divide and Prompt: Chain of Thought Prompting for Text-to-SQLcited by 20
  27. 2022OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Modelscited by 17
  28. 2023Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrievalcited by 16
  29. 2023SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Datacited by 16
  30. 2025STaR-SQL: Self-Taught Reasoner for Text-to-SQLcited by 16
  31. 2021mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformercited by 15
  32. 2025Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errorscited by 14
  33. 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12
  34. 2022BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsingcited by 12
  35. 2023Generating Data for Symbolic Language with Large Language Modelscited by 12
  36. 2023Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chaincited by 12
  37. 2024Using LLM to select the right SQL Query from candidatescited by 12
  38. 2024Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queriescited by 12

50 most cited of 123 All 123 (JSON)