# AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries

arXiv: https://arxiv.org/abs/2406.19073 · HTML: https://mlsanity.com/paper/2406.19073/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [C3: Zero-shot Text-to-SQL with ChatGPT](https://mlsanity.com/paper/2307.07306/) (2023)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries](https://mlsanity.com/paper/2010.11246/) (2020)
- [MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages](https://mlsanity.com/paper/2203.08388/) (2022)
- [ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification](https://mlsanity.com/paper/2310.10996/) (2023)
- [Benchmarking and Improving Text-to-SQL Generation under Ambiguity](https://mlsanity.com/paper/2310.13659/) (2023)
- [Evaluating Cross-Domain Text-to-SQL Models and Benchmarks](https://mlsanity.com/paper/2310.18538/) (2023)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [Learning Executable Semantic Parsers for Natural Language Understanding](https://mlsanity.com/paper/1603.06677/) (2016)
- [Python Code Generation by Asking Clarification Questions](https://mlsanity.com/paper/2212.09885/) (2022)
- [Zero and Few-shot Semantic Parsing with Ambiguous Inputs](https://mlsanity.com/paper/2306.00824/) (2023)
- [AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://mlsanity.com/paper/2302.00762/) (2023)
- [Data Ambiguity Strikes Back: How Documentation Improves GPT's Text-to-SQL](https://mlsanity.com/paper/2310.18742/) (2023)
- [Database Search Results Disambiguation for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2112.08351/) (2021)

## Built on by (most cited first, 30 of 35)

- [PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries](https://mlsanity.com/paper/2410.11076/) (2024)
- [nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoning](https://mlsanity.com/paper/2503.12880/) (2025)
- [BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions](https://mlsanity.com/paper/2510.05318/) (2025)
- [MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.05587/) (2025)
- [Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing](https://mlsanity.com/paper/2502.18448/) (2025)
- [AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL](https://mlsanity.com/paper/2508.15276/) (2025)
- [Asking a Language Model for Diverse Responses](https://mlsanity.com/paper/2509.17570/) (2025)
- [PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL](https://mlsanity.com/paper/2409.14082/) (2024)
- [MathBridge: A Large Corpus Dataset for Translating Spoken Mathematical Expressions into LaTeX Formulas for Improved Readability](https://mlsanity.com/paper/2408.07081/) (2024)
- [BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases](https://mlsanity.com/paper/2505.20321/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types](https://mlsanity.com/paper/2412.17867/) (2024)
- [Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis](https://mlsanity.com/paper/2511.04584/) (2025)
- [LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries](https://mlsanity.com/paper/2601.10398/) (2026)
- [Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study](https://mlsanity.com/paper/2602.12015/) (2026)
- [Language Models Identify Ambiguities and Exploit Loopholes](https://mlsanity.com/paper/2508.19546/) (2025)
- [CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems](https://mlsanity.com/paper/2604.22313/) (2026)
- [SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps](https://mlsanity.com/paper/2510.27532/) (2025)
- [ODUTQA-MDC: A Task for Open-Domain Underspecified Tabular QA with Multi-turn Dialogue-based Clarification](https://mlsanity.com/paper/2604.10159/) (2026)
- [EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement](https://mlsanity.com/paper/2605.00628/) (2026)
- [EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQL](https://mlsanity.com/paper/2605.29670/) (2026)
- [Ambiguity in LLMs is a concept missing problem](https://mlsanity.com/paper/2505.11679/) (2025)
- [Belief Attribution as Mental Explanation: The Role of Accuracy, Informativity, and Causality](https://mlsanity.com/paper/2505.19376/) (2025)
- [RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models](https://mlsanity.com/paper/2505.21409/) (2025)
- [Datrics Text2SQL: A Framework for Natural Language to SQL Query Generation](https://mlsanity.com/paper/2506.12234/) (2025)
- [Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL](https://mlsanity.com/paper/2512.22744/) (2025)
- [PleaSQLarify: Visual Pragmatic Repair for Natural Language Database Querying](https://mlsanity.com/paper/2603.01795/) (2026)
- [ROSE: An Intent-Centered Evaluation Metric for NL2SQL](https://mlsanity.com/paper/2604.12988/) (2026)
- [Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution](https://mlsanity.com/paper/2606.05486/) (2026)
- [SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing](https://mlsanity.com/paper/2606.11424/) (2026)

## Abstract (arXiv)

Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.