# Benchmarking and Improving Text-to-SQL Generation under Ambiguity

arXiv: https://arxiv.org/abs/2310.13659 · HTML: https://mlsanity.com/paper/2310.13659/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://mlsanity.com/paper/2303.13547/) (2023)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [SmBoP: Semi-autoregressive Bottom-up Semantic Parsing](https://mlsanity.com/paper/2010.12412/) (2020)
- [Natural SQL: Making SQL Easier to Infer from Natural Language Specifications](https://mlsanity.com/paper/2109.05153/) (2021)
- [Learning Neural Templates for Text Generation](https://mlsanity.com/paper/1808.10122/) (2018)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback](https://mlsanity.com/paper/2005.02539/) (2020)
- [Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data](https://mlsanity.com/paper/2106.05006/) (2021)
- [Mutual Information and Diverse Decoding Improve Neural Machine Translation](https://mlsanity.com/paper/1601.00372/) (2016)
- [NL-EDIT: Correcting semantic parse errors through natural language interaction](https://mlsanity.com/paper/2103.14540/) (2021)
- [A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation](https://mlsanity.com/paper/2203.15108/) (2022)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [N-Best Hypotheses Reranking for Text-To-SQL Systems](https://mlsanity.com/paper/2210.10668/) (2022)
- [DuoRAT: Towards Simpler Text-to-SQL Models](https://mlsanity.com/paper/2010.11119/) (2020)
- [Interactive-Chain-Prompting: Ambiguity Resolution for Crosslingual Conditional Generation with Interaction](https://mlsanity.com/paper/2301.10309/) (2023)
- [Data Commons](https://mlsanity.com/paper/2309.13054/) (2023)
- [T5QL: Taming language models for SQL generation](https://mlsanity.com/paper/2209.10254/) (2022)
- [Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parsers](https://mlsanity.com/paper/2210.16613/) (2022)
- [Know What I don't Know: Handling Ambiguous and Unanswerable Questions for Text-to-SQL](https://mlsanity.com/paper/2212.08902/) (2022)
- [TempLM: Distilling Language Models into Template-Based Generators](https://mlsanity.com/paper/2205.11055/) (2022)

## Built on by (most cited first, 30 of 34)

- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries](https://mlsanity.com/paper/2406.19073/) (2024)
- [PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries](https://mlsanity.com/paper/2410.11076/) (2024)
- [BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions](https://mlsanity.com/paper/2510.05318/) (2025)
- [Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors](https://mlsanity.com/paper/2501.09310/) (2025)
- [Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing](https://mlsanity.com/paper/2502.18448/) (2025)
- [DocCGen: Document-based Controlled Code Generation](https://mlsanity.com/paper/2406.11925/) (2024)
- [AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL](https://mlsanity.com/paper/2508.15276/) (2025)
- [AmbigNLG: Addressing Task Ambiguity in Instruction for NLG](https://mlsanity.com/paper/2402.17717/) (2024)
- [Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities](https://mlsanity.com/paper/2411.16742/) (2024)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [ODIN: A NL2SQL Recommender to Handle Schema Ambiguity](https://mlsanity.com/paper/2505.19302/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [Mastering the Craft of Data Synthesis for CodeLLMs](https://mlsanity.com/paper/2411.00005/) (2024)
- [Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types](https://mlsanity.com/paper/2412.17867/) (2024)
- [SQaLe: A Large Text-to-SQL Corpus Grounded in Real Schemas](https://mlsanity.com/paper/2602.22223/) (2025)
- [Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence](https://mlsanity.com/paper/2507.10281/) (2025)
- [Evaluating NL2SQL via SQL2NL](https://mlsanity.com/paper/2509.04657/) (2025)
- [Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration](https://mlsanity.com/paper/2510.26495/) (2025)
- [CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems](https://mlsanity.com/paper/2604.22313/) (2026)
- [KeyInst: Keyword Instruction for Improving SQL Formulation in Text-to-SQL](https://mlsanity.com/paper/2411.00788/) (2024)
- [Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation](https://mlsanity.com/paper/2510.09390/) (2025)
- [ODUTQA-MDC: A Task for Open-Domain Underspecified Tabular QA with Multi-turn Dialogue-based Clarification](https://mlsanity.com/paper/2604.10159/) (2026)
- [EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement](https://mlsanity.com/paper/2605.00628/) (2026)
- [EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQL](https://mlsanity.com/paper/2605.29670/) (2026)
- [Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline](https://mlsanity.com/paper/2608.09254/) (2026)
- [AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries?](https://mlsanity.com/paper/2508.09631/) (2025)
- [PleaSQLarify: Visual Pragmatic Repair for Natural Language Database Querying](https://mlsanity.com/paper/2603.01795/) (2026)
- [Reliable Answers for Recurring Questions: Boosting Text-to-SQL Accuracy with Template Constrained Decoding](https://mlsanity.com/paper/2604.28028/) (2026)
- [Evaluating Chinese Ambiguity Understanding in Large Language Models](https://mlsanity.com/paper/2605.15635/) (2026)

## Abstract (arXiv)

Research in Text-to-SQL conversion has been largely benchmarked against datasets where each text query corresponds to one correct SQL. However, natural language queries over real-life databases frequently involve significant ambiguity about the intended SQL due to overlapping schema names and multiple confusing relationship paths. To bridge this gap, we develop a novel benchmark called AmbiQT with over 3000 examples where each text is interpretable as two plausible SQLs due to lexical and/or structural ambiguity. When faced with ambiguity, an ideal top-k decoder should generate all valid interpretations for possible disambiguation by the user. We evaluate several Text-to-SQL systems and decoding algorithms, including those employing state-of-the-art LLMs, and find them to be far from this ideal. The primary reason is that the prevalent beam search algorithm and its variants, treat SQL queries as a string and produce unhelpful token-level diversity in the top-k. We propose LogicalBeam, a new decoding algorithm that navigates the SQL logic space using a blend of plan-based template generation and constrained infilling. Counterfactually generated plans diversify templates while in-filling with a beam-search that branches solely on schema names provides value diversity. LogicalBeam is up to 2.5 times more effective than state-of-the-art models at generating all candidate SQLs in the top-k ranked outputs. It also enhances the top-5 Exact and Execution Match Accuracies on SPIDER and Kaggle DBQA.