# Querying Large Language Models with SQL

arXiv: https://arxiv.org/abs/2304.00472 · HTML: https://mlsanity.com/paper/2304.00472/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 2 Apr 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Break It Down: A Question Understanding Benchmark](https://mlsanity.com/paper/2001.11770/) (2020)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://mlsanity.com/paper/2212.08037/) (2022)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)

## Built on by (most cited first, 14 of 14)

- [UQE: A Query Engine for Unstructured Databases](https://mlsanity.com/paper/2407.09522/) (2024)
- [Research Trends for the Interplay between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2406.08223/) (2024)
- [Hybrid Querying Over Relational Databases and Large Language Models](https://mlsanity.com/paper/2408.00884/) (2024)
- [Large Language Models for Predictive Analysis: How Far Are They?](https://mlsanity.com/paper/2505.17149/) (2025)
- [Unstructured and structured data: Can we have the best of both worlds with large language models?](https://mlsanity.com/paper/2304.13010/) (2023)
- [When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dust](https://mlsanity.com/paper/2305.07230/) (2023)
- [Towards Consistent Language Models Using Declarative Constraints](https://mlsanity.com/paper/2312.15472/) (2023)
- [Contrato360 2.0: A Document and Database-Driven Question-Answer System using Large Language Models and Agents](https://mlsanity.com/paper/2412.17942/) (2024)
- [Recursive Question Understanding for Complex Question Answering over Heterogeneous Personal Data](https://mlsanity.com/paper/2505.11900/) (2025)
- [Implementing Semantic Join Operators Efficiently](https://mlsanity.com/paper/2510.08489/) (2025)
- [Saturn Platform: Foundation Model Operations and Generative AI for Financial Services](https://mlsanity.com/paper/2312.07721/) (2023)
- [RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models](https://mlsanity.com/paper/2505.21409/) (2025)
- [Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning](https://mlsanity.com/paper/2607.22621/) (2026)
- [LYNX: Analytical Program Synthesis via Grounding and Closed-Loop Refinement](https://mlsanity.com/paper/2610.14777/) (2026)

## Abstract (arXiv)

In many use-cases, information is stored in text but not available in structured data. However, extracting data from natural language text to precisely fit a schema, and thus enable querying, is a challenging task. With the rise of pre-trained Large Language Models (LLMs), there is now an effective solution to store and use information extracted from massive corpora of text documents. Thus, we envision the use of SQL queries to cover a broad range of data that is not captured by traditional databases by tapping the information in LLMs. To ground this vision, we present Galois, a prototype based on a traditional database architecture, but with new physical operators for querying the underlying LLM. The main idea is to execute some operators of the the query plan with prompts that retrieve data from the LLM. For a large class of SQL queries, querying LLMs returns well structured relations, with encouraging qualitative results. Preliminary experimental results make pre-trained LLMs a promising addition to the field of database systems, introducing a new direction for hybrid query processing. However, we pinpoint several research challenges that must be addressed to build a DBMS that exploits LLMs. While some of these challenges necessitate integrating concepts from the NLP literature, others offer novel research avenues for the DB community.