mlsanityData to 14 Oct 2026

SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models

arXiv 2403.02951 ↗First posted 5 Mar 2024cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods.Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions. To address the aforementioned issues, we firstly construct a new dataset designed to mitigate the risk of overfitting in LLMs. Then we formulate five evaluation tasks to comprehensively assess the performance of diverse methods across various LLMs throughout the Text-to-SQL process.Our study highlights the performance disparities among LLMs and proposes optimal in-context learning solutions tailored to each task. These findings offer valuable insights for facilitating the development of LLM-based Text-to-SQL systems.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

43 of 60 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  10. 2023Teaching Large Language Models to Self-Debugcited by 988
  11. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  12. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
31 more
  1. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  2. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  3. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  4. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  5. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
  6. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 164
  7. 2017SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learningcited by 135
  8. 2020Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsingcited by 133
  9. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  10. 2021LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relationscited by 119
  11. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 119
  12. 2018Coarse-to-Fine Decoding for Neural Semantic Parsingcited by 110
  13. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  14. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  15. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  16. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  17. 2019A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualizationcited by 88
  18. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  19. 2020RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databasescited by 63
  20. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 52
  21. 2021SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQLcited by 43
  22. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
  23. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  24. 2020Hybrid Ranking Network for Text-to-SQLcited by 33
  25. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  26. 2018SQL-to-Text Generation with Graph-to-Sequence Modelcited by 31
  27. 2021Logic-Consistency Text Generation from Semantic Parsescited by 23
  28. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  29. 2023Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrievalcited by 16
  30. 2022Deep Learning Driven Natural Languages Text to SQL Query Conversion: A Surveycited by 11
  31. 2023Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function -- with Real Applications in Traffic Domaincited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 5.

32 papers.

Sort:

By area:

  1. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  2. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  3. 2024MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQLcited by 35
  4. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 33
  5. 2024PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistencycited by 27
  6. 2024DMQR-RAG: Diverse Multi-Query Rewriting for RAGcited by 27
  7. 2025Agentar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scalingcited by 24
  8. 2024From Natural Language to SQL: Review of LLM-based Text-to-SQL Systemscited by 16
  9. 2024You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQLcited by 11
  10. 2024FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formatscited by 8
  11. 2024PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQLcited by 7
  12. 2024Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systemscited by 6
20 more
  1. 2024DAC: Decomposed Automation Correction for Text-to-SQLcited by 6
  2. 2024TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Modelscited by 5
  3. 2026EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolutioncited by 5
  4. 2025Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunitiescited by 4
  5. 2024Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurementcited by 3
  6. 2025MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstractioncited by 3
  7. 2026Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibilitycited by 3
  8. 2024Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQLcited by 2
  9. 2025Independent Mobility GPT (IDM-GPT): A Self-Supervised Multi-Agent Large Language Model Framework for Customized Traffic Mobility Analysis Using Machine Learning Modelscited by 2
  10. 2025LiLM-RDB-SFC: Lightweight Language Model with Relational Database-Guided DRL for Optimized SFC Provisioningcited by 2
  11. 2025End-to-End Text-to-SQL with Dataset Selection: Leveraging LLMs for Adaptive Query Generationcited by 2
  12. 2025Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Explorationcited by 2
  13. 2026Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoningcited by 2
  14. 2026From Natural Language to Executable Option Strategies via Large Language Modelscited by 1
  15. 2025Meta-aware Learning in text-to-SQL Large Language Modelcited by 0
  16. 2025Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Modelscited by 0
  17. 2025SPORTSQL: An Interactive System for Real-Time Sports Reasoning and Visualizationcited by 0
  18. 2025LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQLcited by 0
  19. 2025GPG: Generalized Policy Gradient Theorem for Transformer-based Policiescited by 0
  20. 2026Vibe Coding on Trial: Operating Characteristics of Unanimous LLM Juriescited by 0