# Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation

arXiv: https://arxiv.org/abs/2306.08891 · HTML: https://mlsanity.com/paper/2306.08891/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 52 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)

## Built on by (most cited first, 24 of 24)

- [CodeS: Towards Building Open-source Language Models for Text-to-SQL](https://mlsanity.com/paper/2402.16347/) (2024)
- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [PURPLE: Making a Large Language Model a Better SQL Writer](https://mlsanity.com/paper/2403.20014/) (2024)
- [Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey](https://mlsanity.com/paper/2310.17894/) (2023)
- [Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation](https://mlsanity.com/paper/2402.17144/) (2024)
- [ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalization](https://mlsanity.com/paper/2405.06683/) (2024)
- [ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL](https://mlsanity.com/paper/2412.10138/) (2024)
- [TrustUQA: A Trustful Framework for Unified Structured Data Question Answering](https://mlsanity.com/paper/2406.18916/) (2024)
- [SEED: Domain-Specific Data Curation With Large Language Models](https://mlsanity.com/paper/2310.00749/) (2023)
- [DBCopilot: Natural Language Querying over Massive Databases via Schema Routing](https://mlsanity.com/paper/2312.03463/) (2023)
- [Grounding Natural Language to SQL Translation with Data-Based Self-Explanations](https://mlsanity.com/paper/2411.02948/) (2024)
- [DAC: Decomposed Automation Correction for Text-to-SQL](https://mlsanity.com/paper/2408.08779/) (2024)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL](https://mlsanity.com/paper/2404.02389/) (2024)
- [Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence](https://mlsanity.com/paper/2507.10281/) (2025)
- [Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning](https://mlsanity.com/paper/2404.13077/) (2024)
- [HES-SQL: Hybrid Reasoning for Efficient Text-to-SQL with Structural Skeleton Guidance](https://mlsanity.com/paper/2510.08896/) (2025)
- [ORANGE: An Online Reflection ANd GEneration framework with Domain Knowledge for Text-to-SQL](https://mlsanity.com/paper/2511.00985/) (2025)
- [Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation](https://mlsanity.com/paper/2511.13590/) (2025)
- [PLM4NDV: Minimizing Data Access for Number of Distinct Values Estimation with Pre-trained Language Models](https://mlsanity.com/paper/2504.00608/) (2025)
- [A Multi-agent Text2SQL Framework using Small Language Models and Execution Feedback](https://mlsanity.com/paper/2512.18622/) (2025)
- [Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL](https://mlsanity.com/paper/2512.22744/) (2025)
- [When Prompts Become Payloads: A Framework for Mitigating SQL Injection Attacks in Large Language Model-Driven Applications](https://mlsanity.com/paper/2605.10176/) (2026)
- [SafeQL: Search-based Refinement for Safe and Efficient LLM-based Text-to-SQL](https://mlsanity.com/paper/2608.09260/) (2026)

## Abstract (arXiv)

Zero-shot NL2SQL is crucial in achieving natural language to SQL that is adaptive to new environments (e.g., new databases, new linguistic phenomena or SQL structures) with zero annotated NL2SQL samples from such environments. Existing approaches either fine-tune pre-trained language models (PLMs) based on annotated data or use prompts to guide fixed large language models (LLMs) such as ChatGPT. PLMs can perform well in schema alignment but struggle to achieve complex reasoning, while LLMs is superior in complex reasoning tasks but cannot achieve precise schema alignment. In this paper, we propose a ZeroNL2SQL framework that combines the complementary advantages of PLMs and LLMs for supporting zero-shot NL2SQL. ZeroNL2SQL first uses PLMs to generate an SQL sketch via schema alignment, then uses LLMs to fill the missing information via complex reasoning. Moreover, in order to better align the generated SQL queries with values in the given database instances, we design a predicate calibration method to guide the LLM in completing the SQL sketches based on the database instances and select the optimal SQL query via an execution-based strategy. Comprehensive experiments show that ZeroNL2SQL can achieve the best zero-shot NL2SQL performance on real-world benchmarks. Specifically, ZeroNL2SQL outperforms the state-of-the-art PLM-based methods by 3.2% to 13% and exceeds LLM-based methods by 10% to 20% on execution accuracy.