mlsanityData to 7 Oct 2026

ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems

arXiv 2306.04743 ↗First posted 7 Jun 2023cs.DB cs.AI cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

Natural Language to SQL systems (NL-to-SQL) have recently shown a significant increase in accuracy for natural language to SQL query translation. This improvement is due to the emergence of transformer-based language models, and the popularity of the Spider benchmark - the de-facto standard for evaluating NL-to-SQL systems. The top NL-to-SQL systems reach accuracies of up to 85%. However, Spider mainly contains simple databases with few tables, columns, and entries, which does not reflect a realistic setting. Moreover, complex real-world databases with domain-specific content have little to no training data available in the form of NL/SQL-pairs leading to poor performance of existing NL-to-SQL systems. In this paper, we introduce ScienceBenchmark, a new complex NL-to-SQL benchmark for three real-world, highly domain-specific databases. For this new benchmark, SQL experts and domain experts created high-quality NL/SQL-pairs for each domain. To garner more data, we extended the small amount of human-generated data with synthetic data generated using GPT-3. We show that our benchmark is highly challenging, as the top performing systems on Spider achieve a very low performance on our benchmark. Thus, the challenge is many-fold: creating NL-to-SQL systems for highly complex domains with a small amount of hand-made training data augmented with synthetic data. To our knowledge, ScienceBenchmark is the first NL-to-SQL benchmark designed with complex real-world scientific databases, containing challenging training and test data carefully validated by domain experts.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

28 of 52 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  8. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  9. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  10. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  11. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  12. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
16 more
  1. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  2. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  3. 2017SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learningcited by 135
  4. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  5. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  6. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  7. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  8. 2019A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualizationcited by 88
  9. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 76
  10. 2021Natural SQL: Making SQL Easier to Infer from Natural Language Specificationscited by 66
  11. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  12. 2019A Comparative Survey of Recent Natural Language Interfaces for Databasescited by 33
  13. 2021Data Augmentation with Hierarchical SQL-to-Question Generation for Cross-domain Text-to-SQL Parsingcited by 32
  14. 2018Question Generation from SQL Queries Improves Neural Semantic Parsingcited by 29
  15. 2020A Methodology for Creating Question Answering Corpora Using Inverse Data Annotationcited by 5
  16. 2021INODE: Building an End-to-End Data Exploration System in Practice [Extended Vision]cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 19; 2026: 11.

42 papers.

Sort:

By area:

  1. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  2. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  3. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  4. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  5. 2024Large Language Model for Table Processing: A Surveycited by 52
  6. 2025Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQLcited by 38
  7. 2024BEAVER: An Enterprise Benchmark for Text-to-SQLcited by 37
  8. 2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 25
  9. 2024Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translationcited by 17
  10. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  11. 2025Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errorscited by 14
  12. 2024Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queriescited by 12
30 more
  1. 2024SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmarkcited by 10
  2. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challengescited by 10
  3. 2024Grounding Natural Language to SQL Translation with Data-Based Self-Explanationscited by 7
  4. 2024Multi-Modal Data Exploration via Language Agentscited by 7
  5. 2024StatBot.Swiss: Bilingual Open Data Exploration in Natural Languagecited by 6
  6. 2025SPFT-SQL: Enhancing Large Language Model for Text-to-SQL Parsing by Self-Play Fine-Tuningcited by 5
  7. 2025Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunitiescited by 4
  8. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  9. 2025STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databasescited by 2
  10. 2025Exploring Database Normalization Effects on SQL Generationcited by 2
  11. 2026An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Datacited by 2
  12. 2025REFORMER: A ChatGPT-Driven Data Synthesis Framework Elevating Text-to-SQL Modelscited by 1
  13. 2025LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialoguescited by 1
  14. 2025DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extractioncited by 1
  15. 2025ORANGE: An Online Reflection ANd GEneration framework with Domain Knowledge for Text-to-SQLcited by 1
  16. 2025Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translationcited by 1
  17. 2026EvolSQL: Structure-Aware Evolution for Scalable Text-to-SQL Data Synthesiscited by 1
  18. 2026NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutionscited by 1
  19. 2026R3-SQL: Ranking Reward and Resampling for Text-to-SQLcited by 1
  20. 2026SQLConductor: Search-to-Policy Learning for Step-wise Text-to-SQL Orchestrationcited by 1
  21. 2025GradeSQL: Test-Time Inference with Outcome Reward Models for Text-to-SQL Generation from Large Language Modelscited by 0
  22. 2025Skeletons Matter: Dynamic Data Augmentation for Text-to-Querycited by 0
  23. 2025Query Carefully: Detecting the Unanswerables in Text-to-SQL Taskscited by 0
  24. 2025Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQLcited by 0
  25. 2026SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesiscited by 0
  26. 2026A Systematic Approach for Large Language Models Debuggingcited by 0
  27. 2026Knowledge Distillation for Low-Resource Open-source Text-to-SQL Modelcited by 0
  28. 2026STRATOS: Bridging the Symbolic-to-Numeric Gap in Spatio-Temporal Text-to-SQL for Meteorological Datacited by 0
  29. 2026COAL-SQL: Coverage-Guided Augmentation and Failure-Driven Learning for Text-to-SQL Post-Trainingcited by 0
  30. 2026TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databasescited by 0