mlsanityData to 14 Oct 2026

Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT -- A Text-to-SQL Parsing Comparison

arXiv 2310.10190 ↗First posted 16 Oct 2023cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

The success of ChatGPT has ignited an AI race, with researchers striving to develop new large language models (LLMs) that can match or surpass the language understanding and generation abilities of commercial ones. In recent times, a number of models have emerged, claiming performance near that of GPT-3.5 or GPT-4 through various instruction-tuning methods. As practitioners of Text-to-SQL parsing, we are grateful for their valuable contributions to open-source research. However, it is important to approach these claims with a sense of scrutiny and ascertain the actual effectiveness of these models. Therefore, we pit six popular large language models against each other, systematically evaluating their Text-to-SQL parsing capability on nine benchmark datasets with five different prompting strategies, covering both zero-shot and few-shot scenarios. Regrettably, the open-sourced models fell significantly short of the performance achieved by closed-source models like GPT-3.5, highlighting the need for further work to bridge the performance gap between these models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

32 of 47 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
20 more
  1. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  2. 2022Data Contamination: From Memorization to Exploitationcited by 236
  3. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
  4. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
  5. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  6. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  7. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  8. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 119
  9. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  10. 2017Learning a Neural Semantic Parser from User Feedbackcited by 99
  11. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 70
  12. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  13. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 68
  14. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 58
  15. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 45
  16. 2023Improving Generalization in Language Model-Based Text-to-SQL Semantic Parsing: Two Simple Semantic Boundary-Based Techniquescited by 24
  17. 2023Divide and Prompt: Chain of Thought Prompting for Text-to-SQLcited by 20
  18. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  19. 2023UNITE: A Unified Benchmark for Text-to-SQL Evaluationcited by 14
  20. 2017Type- and Content-Driven Synthesis of SQL Queries from Natural Languagecited by 5

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 1.

6 papers.

  1. 2024Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environmentscited by 39
  2. 2024SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaborationcited by 11
  3. 2025Linguistic Blind Spots of Large Language Modelscited by 6
  4. 2024From Text to CQL: Bridging Natural Language and Corpus Search Enginecited by 1
  5. 2024An Actor-Critic Approach to Boosting Text-to-SQL Large Language Modelcited by 1
  6. 2024A Survey on the Real Power of ChatGPTcited by 0