mlsanityData to 7 Oct 2026

Evaluating the Text-to-SQL Capabilities of Large Language Models

arXiv 2204.00498 ↗First posted 15 Mar 2022cs.CL cs.DB cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 10 of its 23 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

10 of 23 references.

References · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  4. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  5. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  6. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  7. 2020Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsingcited by 133
  8. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  9. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  10. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 36; 2024: 21; 2025: 24; 2026: 9.

101 papers.

Sort:

By area:

  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  3. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  4. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  5. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  6. 2022Binding Language Models in Symbolic Languagescited by 224
  7. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  8. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
  9. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  10. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  11. 2022Learning by Distilling Contextcited by 128
  12. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 118
38 more
  1. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  2. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  3. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  4. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  5. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  6. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 61
  7. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  8. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 57
  9. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  10. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  11. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  12. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  13. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 50
  14. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  15. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  16. 2023Large Language Models and Simple, Stupid Bugscited by 45
  17. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
  18. 2023SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)cited by 42
  19. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  20. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
  21. 2024TableRAG: Million-Token Table Understanding with Language Modelscited by 40
  22. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  23. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 32
  24. 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
  25. 2023Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generationcited by 24
  26. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  27. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  28. 2024H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tablescited by 21
  29. 2023Divide and Prompt: Chain of Thought Prompting for Text-to-SQLcited by 20
  30. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understandingcited by 20
  31. 2024Interpretable LLM-based Table Question Answeringcited by 18
  32. 2025TableMaster: A Recipe to Advance Table Understanding with Language Modelscited by 18
  33. 2024Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translationcited by 17
  34. 2023Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrievalcited by 16
  35. 2025STaR-SQL: Self-Taught Reasoner for Text-to-SQLcited by 16
  36. 2024From Natural Language to SQL: Review of LLM-based Text-to-SQL Systemscited by 15
  37. 2023UNITE: A Unified Benchmark for Text-to-SQL Evaluationcited by 14
  38. 2024ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQLcited by 13

50 most cited of 101 All 101 (JSON)