mlsanityData to 14 Oct 2026

Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

arXiv 2305.03111 ↗First posted 4 May 2023cs.CLcited by 688

Not on the map. Why?

Cited by 134 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-to-SQL parsing, which aims at converting natural language instructions into executable SQLs, has gained increasing attention in recent years. In particular, Codex and ChatGPT have shown impressive results in this task. However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications. To mitigate this gap, we present Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents, and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL models must feature database value comprehension in addition to semantic parsing. The experimental results demonstrate the significance of database values in generating accurate text-to-SQLs for big databases. Furthermore, even the most effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand. Besides, we also provide an efficiency analysis to offer insights into generating text-to-efficient-SQLs that are beneficial to industries. We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research. The leaderboard and source code are available: https://bird-bench.github.io/.

Key facts. Cited by 688 papers in the map (data through 14 Oct 2026). Builds on 45 of its 71 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

45 of 71 references.

References · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  12. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
33 more
  1. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  2. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  3. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  4. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  5. 2023Augmented Language Models: a Surveycited by 403
  6. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  7. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  8. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  9. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  10. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  11. 2017SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learningcited by 135
  12. 2021LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relationscited by 119
  13. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  14. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  15. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  16. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  17. 2017Learning a Neural Semantic Parser from User Feedbackcited by 99
  18. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  19. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  20. 2018TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generationcited by 81
  21. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 77
  22. 2020Grounded Adaptation for Zero-shot Executable Semantic Parsingcited by 76
  23. 2019Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsingcited by 74
  24. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  25. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 52
  26. 2023EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Recordscited by 45
  27. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  28. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  29. 2021Improving Text-to-SQL with Schema Dependency Learningcited by 25
  30. 2017An Encoder-Decoder Framework Translating Natural Language to Database Queriescited by 23
  31. 2022STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsingcited by 19
  32. 2023Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledgecited by 18
  33. 2020Batchwise Probabilistic Incremental Data Cleaningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 34; 2024: 150; 2025: 224; 2026: 280.

688 papers.

Sort:

By area:

  1. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  5. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  6. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  7. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  8. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  10. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  11. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  12. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
38 more
  1. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  2. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  3. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
  4. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  5. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
  6. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  7. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 128
  8. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  9. 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 105
  10. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  11. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  12. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  13. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  14. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  15. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  16. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 72
  17. 2026Qwen3-Coder-Next Technical Reportcited by 71
  18. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  19. 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
  20. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  21. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  22. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  23. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  24. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 56
  25. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  26. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 54
  27. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 54
  28. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  29. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  30. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 53
  31. 2024Large Language Model for Table Processing: A Surveycited by 52
  32. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 52
  33. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 50
  34. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 50
  35. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 47
  36. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 47
  37. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 47
  38. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46

50 most cited of 688 All 688 (JSON)