mlsanityData to 14 Oct 2026

A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

arXiv 2303.13547 ↗First posted 12 Mar 2023cs.CL cs.AIcited by 119

Not on the map. Why?

Abstract (arXiv)

This paper presents the first comprehensive analysis of ChatGPT's Text-to-SQL ability. Given the recent emergence of large-scale conversational language model ChatGPT and its impressive capabilities in both conversational abilities and code generation, we sought to evaluate its Text-to-SQL performance. We conducted experiments on 12 benchmark datasets with different languages, settings, or scenarios, and the results demonstrate that ChatGPT has strong text-to-SQL abilities. Although there is still a gap from the current state-of-the-art (SOTA) model performance, considering that the experiment was conducted in a zero-shot scenario, ChatGPT's performance is still impressive. Notably, in the ADVETA (RPL) scenario, the zero-shot ChatGPT even outperforms the SOTA model that requires fine-tuning on the Spider dataset by 4.1%, demonstrating its potential for use in practical applications. To support further research in related fields, we have made the data generated by ChatGPT publicly available at https://github.com/THU-BPM/chatgpt-sql.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

26 of 30 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  6. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  7. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  8. 2023Mathematical Capabilities of ChatGPTcited by 268
  9. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  10. 2022Binding Language Models in Symbolic Languagescited by 228
  11. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
  12. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
14 more
  1. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  2. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  3. 2021LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relationscited by 119
  4. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  5. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  6. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  7. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  8. 2019A Pilot Study for Chinese SQL Semantic Parsingcited by 33
  9. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  10. 2022Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbationcited by 28
  11. 2022Semantic Enhanced Text-to-SQL Parsing via Iteratively Learning Schema Linking Graphcited by 22
  12. 2022STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsingcited by 19
  13. 2022Measuring and Improving Compositional Generalization in Text-to-SQL via Component Alignmentcited by 16
  14. 2022CQR-SQL: Conversational Question Reformulation Enhanced Context-Dependent Text-to-SQL Parserscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 47; 2024: 45; 2025: 15; 2026: 12.

119 papers.

Sort:

By area:

  1. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  2. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  3. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 343
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  5. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  6. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 164
  7. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  8. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  9. 2023A Semantic Invariant Robust Watermark for Large Language Modelscited by 112
  10. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
  11. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  12. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 77
38 more
  1. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  2. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  3. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  4. 2023An Unforgeable Publicly Verifiable Watermark for Large Language Modelscited by 65
  5. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  6. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 58
  7. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  8. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 45
  9. 2023SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)cited by 42
  10. 2023Exploring the Potential of Large Language Models in Computational Argumentationcited by 42
  11. 2024Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLMcited by 35
  12. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 34
  13. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  14. 2023Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistancecited by 33
  15. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  16. 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
  17. 2024Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigmcited by 31
  18. 2025LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQLcited by 29
  19. 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
  20. 2024Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrievalcited by 27
  21. 2024OpenTab: Advancing Large Language Models as Open-domain Table Reasonerscited by 25
  22. 2023Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generationcited by 24
  23. 2023CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQLcited by 24
  24. 2024BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebracited by 24
  25. 2023Xpert: Empowering Incident Management with Query Recommendations via Large Language Modelscited by 22
  26. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  27. 2023Divide and Prompt: Chain of Thought Prompting for Text-to-SQLcited by 20
  28. 2023SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understandingcited by 18
  29. 2023Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidencecited by 17
  30. 2023Specialist or Generalist? Instruction Tuning for Specific NLP Taskscited by 17
  31. 2024Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translationcited by 17
  32. 2025STaR-SQL: Self-Taught Reasoner for Text-to-SQLcited by 17
  33. 2023Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrievalcited by 16
  34. 2023SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Datacited by 16
  35. 2024Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruningcited by 16
  36. 2023Semantic Decomposition of Question and SQL for Text-to-SQL Parsingcited by 14
  37. 2024ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQLcited by 13
  38. 2023Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chaincited by 12

50 most cited of 119 All 119 (JSON)