mlsanityData to 14 Oct 2026

Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation

arXiv 2308.15363 ↗First posted 29 Aug 2023cs.DB cs.CL cs.LGcited by 385

Not on the map. Why?

Cited by 55 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.

Key facts. Cited by 385 papers in the map (data through 14 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 29 Aug 2023.

Builds on

44 of 68 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
32 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  4. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  5. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  6. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  7. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  8. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  9. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  10. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  11. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
  12. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 164
  13. 2020Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsingcited by 133
  14. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  15. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  16. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 119
  17. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  18. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  19. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  20. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  21. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 58
  22. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 52
  23. 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
  24. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 45
  25. 2022S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parserscited by 41
  26. 2018SQL-to-Text Generation with Graph-to-Sequence Modelcited by 31
  27. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  28. 2022HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsingcited by 26
  29. 2017An Encoder-Decoder Framework Translating Natural Language to Database Queriescited by 23
  30. 2022Natural Language Interfaces to Datacited by 14
  31. 2023From Words to Code: Harnessing Data for Program Synthesis from Natural Languagecited by 11
  32. 2020Duoquest: A Dual-Specification System for Expressive SQL Queriescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 104; 2025: 151; 2026: 123.

385 papers.

Sort:

By area:

  1. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  2. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  3. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
  4. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  5. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
  6. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  7. 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 105
  8. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  9. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  10. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  11. 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
  12. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
38 more
  1. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  2. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  3. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 56
  4. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 54
  5. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 53
  6. 2024Large Language Model for Table Processing: A Surveycited by 52
  7. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 52
  8. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 50
  9. 2024DTS-SQL: Decomposed Text-to-SQL with Small Large Language Modelscited by 45
  10. 2024BEAVER: An Enterprise Benchmark for Text-to-SQLcited by 42
  11. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  12. 2025XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQLcited by 40
  13. 2024Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environmentscited by 39
  14. 2025Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQLcited by 38
  15. 2025ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Explorationcited by 37
  16. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  17. 2024Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLMcited by 35
  18. 2024MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQLcited by 35
  19. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 35
  20. 2024E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQLcited by 35
  21. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  22. 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
  23. 2024Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigmcited by 31
  24. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
  25. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  26. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
  27. 2024A Survey of Useful LLM Evaluationcited by 29
  28. 2024PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistencycited by 27
  29. 2024Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrievalcited by 27
  30. 2024BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebracited by 24
  31. 2025ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedbackcited by 24
  32. 2024MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQLcited by 22
  33. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 22
  34. 2024Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translationcited by 21
  35. 2024UQE: A Query Engine for Unstructured Databasescited by 21
  36. 2024Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Waycited by 21
  37. 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 20
  38. 2024MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translationcited by 19

50 most cited of 385 All 385 (JSON)