mlsanityData to 7 Oct 2026

CodeS: Towards Building Open-source Language Models for Text-to-SQL

arXiv 2402.16347 ↗First posted 26 Feb 2024cs.CL cs.DBcited by 183

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models have shown promising performance on the task of translating natural language questions into SQL queries (Text-to-SQL). However, most of the state-of-the-art (SOTA) approaches rely on powerful yet closed-source large language models (LLMs), such as ChatGPT and GPT-4, which may have the limitations of unclear model architectures, data privacy risks, and expensive inference overheads. To address the limitations, we introduce CodeS, a series of pre-trained language models with parameters ranging from 1B to 15B, specifically designed for the text-to-SQL task. CodeS is a fully open-source language model, which achieves superior accuracy with much smaller parameter sizes. This paper studies the research challenges in building CodeS. To enhance the SQL generation abilities of CodeS, we adopt an incremental pre-training approach using a specifically curated SQL-centric corpus. Based on this, we address the challenges of schema linking and rapid domain adaptation through strategic prompt construction and a bi-directional data augmentation technique. We conduct comprehensive evaluations on multiple datasets, including the widely used Spider benchmark, the newly released BIRD benchmark, robustness-diagnostic benchmarks such as Spider-DK, Spider-Syn, Spider-Realistic, and Dr.Spider, as well as two real-world datasets created for financial and academic applications. The experimental results show that our CodeS achieves new SOTA accuracy and robustness on nearly all challenging text-to-SQL benchmarks.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Builds on 59 of its 87 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

59 of 87 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  11. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  12. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  13. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  14. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  15. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  16. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  17. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  18. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  19. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  20. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  21. 2022Active Example Selection for In-Context Learningcited by 195
  22. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  23. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  24. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  25. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  26. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  27. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  28. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  29. 2021LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relationscited by 119
  30. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 118
  31. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  32. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  33. 2020SmBoP: Semi-autoregressive Bottom-up Semantic Parsingcited by 97
  34. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  35. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  36. 2018SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Taskcited by 74
  37. 2020Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Trainingcited by 65
  38. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 57

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 84; 2026: 57.

183 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  3. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
  4. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 132
  5. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  6. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  7. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  8. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 72
  9. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  10. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  11. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  12. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
38 more
  1. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 51
  2. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 51
  3. 2025XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQLcited by 40
  4. 2024E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQLcited by 35
  5. 2024MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQLcited by 34
  6. 2025ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Explorationcited by 34
  7. 2024MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQLcited by 22
  8. 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 19
  9. 2025SQL-o1: A Self-Reward Heuristic Dynamic Search Method for Text-to-SQLcited by 19
  10. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
  11. 2024MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translationcited by 17
  12. 2024TQA-Bench: Evaluating LLMs for Multi-Table Question Answeringcited by 17
  13. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  14. 2024SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Mergingcited by 16
  15. 2024From Natural Language to SQL: Review of LLM-based Text-to-SQL Systemscited by 15
  16. 2025ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelinescited by 15
  17. 2025CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learningcited by 15
  18. 2025SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applicationscited by 15
  19. 2025AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scalecited by 15
  20. 2025Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errorscited by 14
  21. 2025Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQLcited by 14
  22. 2025MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQLcited by 14
  23. 2024ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQLcited by 13
  24. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 12
  25. 2024You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQLcited by 11
  26. 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
  27. 2024Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQLcited by 11
  28. 2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 11
  29. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
  30. 2024SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaborationcited by 10
  31. 2025SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQLcited by 10
  32. 2025SLM-SQL: An Exploration of Small Language Models for Text-to-SQLcited by 10
  33. 2025MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQLcited by 10
  34. 2024E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Modelcited by 9
  35. 2025BASE-SQL: A powerful open source Text-To-SQL baseline approachcited by 9
  36. 2025Knapsack Optimization-based Schema Linking for LLM-based Text-to-SQL Generationcited by 9
  37. 2025ReEx-SQL: Reasoning with Execution-Aware Reinforcement Learning for Text-to-SQLcited by 8
  38. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7

50 most cited of 183 All 183 (JSON)