mlsanityData to 14 Oct 2026

KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

arXiv 2106.11455 ↗First posted 22 Jun 2021cs.CL cs.AI cs.DBcited by 106

Not on the map. Why?

Abstract (arXiv)

The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

22 of 29 references.

References · 22

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  3. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  4. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  5. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  6. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  7. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  8. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  9. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  10. 2019Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questionscited by 80
  11. 2020Learning from Task Descriptionscited by 78
  12. 2019Code Generation as a Dual Task of Code Summarizationcited by 62
10 more
  1. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  2. 2020Meta-Learning for Domain Generalization in Semantic Parsingcited by 41
  3. 2018Natural Language to Structured Query Generation via Meta-Learningcited by 33
  4. 2020Photon: A Robust Cross-Domain Text-to-SQL Systemcited by 26
  5. 2016Learning Executable Semantic Parsers for Natural Language Understandingcited by 21
  6. 2020Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQLcited by 16
  7. 2019Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learningcited by 10
  8. 2021Few-Shot Semantic Parsing for New Predicatescited by 7
  9. 2019Associating Natural Language Comment and Source Code Entitiescited by 6
  10. 2020Domain Adaptation for Semantic Parsingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 21; 2024: 28; 2025: 27; 2026: 20.

106 papers.

Sort:

By area:

  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  2. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  3. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  4. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  5. 2021Meta-learning via Language Model In-context Tuningcited by 164
  6. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  8. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  9. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  10. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 56
  11. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  12. 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
38 more
  1. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  2. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 46
  3. 2023EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Recordscited by 45
  4. 2023ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systemscited by 45
  5. 2024BEAVER: An Enterprise Benchmark for Text-to-SQLcited by 42
  6. 2024FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysiscited by 36
  7. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 35
  8. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 34
  9. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 34
  10. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  11. 2023MultiTabQA: Generating Tabular Answers for Multi-Table Question Answeringcited by 32
  12. 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
  13. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
  14. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
  15. 2024Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrievalcited by 27
  16. 2023A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databasescited by 25
  17. 2023Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generationcited by 24
  18. 2022Knowledge Base Question Answering: A Semantic Parsing Perspectivecited by 22
  19. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  20. 2024PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queriescited by 19
  21. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 19
  22. 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 17
  23. 2024From Natural Language to SQL: Review of LLM-based Text-to-SQL Systemscited by 16
  24. 2021Weakly Supervised Text-to-SQL Parsing through Question Decompositioncited by 15
  25. 2023UNITE: A Unified Benchmark for Text-to-SQL Evaluationcited by 14
  26. 2024SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmarkcited by 14
  27. 2025Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQLcited by 14
  28. 2025MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmarkcited by 14
  29. 2024Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoningcited by 13
  30. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 13
  31. 2023Error Detection for Text-to-SQL Semantic Parsingcited by 12
  32. 2024Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queriescited by 12
  33. 2023From Words to Code: Harnessing Data for Program Synthesis from Natural Languagecited by 11
  34. 2024You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQLcited by 11
  35. 2025Confidence Estimation for Error Detection in Text-to-SQL Systemscited by 11
  36. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  37. 2023SEED: Domain-Specific Data Curation With Large Language Modelscited by 10
  38. 2023LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?cited by 10

50 most cited of 106 All 106 (JSON)