# End-to-end Text-to-SQL Generation within an Analytics Insight Engine

arXiv: https://arxiv.org/abs/2406.12104 · HTML: https://mlsanity.com/paper/2406.12104/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL](https://mlsanity.com/paper/2312.11242/) (2023)
- [SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning](https://mlsanity.com/paper/1711.04436/) (2017)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation](https://mlsanity.com/paper/2405.07467/) (2024)
- [SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task](https://mlsanity.com/paper/1810.05237/) (2018)
- [Robust Text-to-SQL Generation with Execution-Guided Decoding](https://mlsanity.com/paper/1807.03100/) (2018)
- [Semantic Parsing with Syntax- and Table-Aware SQL Generation](https://mlsanity.com/paper/1804.08338/) (2018)

## Built on by (most cited first, 7 of 7)

- [The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models](https://mlsanity.com/paper/2408.07702/) (2024)
- [Towards Optimizing SQL Generation via LLM Routing](https://mlsanity.com/paper/2411.04319/) (2024)
- [Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL](https://mlsanity.com/paper/2411.13244/) (2024)
- [GenEdit: Compounding Operators and Continuous Improvement to Tackle Text-to-SQL in the Enterprise](https://mlsanity.com/paper/2503.21602/) (2025)
- [STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases](https://mlsanity.com/paper/2509.19508/) (2025)
- [NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions](https://mlsanity.com/paper/2604.16493/) (2026)
- [SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation](https://mlsanity.com/paper/2609.08950/) (2026)

## Abstract (arXiv)

Recent advancements in Text-to-SQL have pushed database management systems towards greater democratization of data access. Today's language models are at the core of these advancements. They enable impressive Text-to-SQL generation as experienced in the development of Distyl AI's Analytics Insight Engine. Its early deployment with enterprise customers has highlighted three core challenges. First, data analysts expect support with authoring SQL queries of very high complexity. Second, requests are ad-hoc and, as such, require low latency. Finally, generation requires an understanding of domain-specific terminology and practices. The design and implementation of our Text-to-SQL generation pipeline, powered by large language models, tackles these challenges. The core tenants of our approach rely on external knowledge that we extract in a pre-processing phase, on retrieving the appropriate external knowledge at query generation time, and on decomposing SQL query generation following a hierarchical CTE-based structure. Finally, an adaptation framework leverages feedback to update the external knowledge, in turn improving query generation over time. We give an overview of our end-to-end approach and highlight the operators generating SQL during inference.