mlsanityData to 7 Oct 2026

AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries

arXiv 2406.19073 ↗First posted 27 Jun 2024cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 23 of its 49 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

23 of 49 references.

References · 23

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  4. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  5. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  6. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  7. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  8. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  9. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  10. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  11. 2022POTATO: The Portable Text Annotation Toolcited by 64
  12. 2020On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queriescited by 59
11 more
  1. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  2. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 37
  3. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  4. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 31
  5. 2023A Survey on Asking Clarification Questions Datasets in Conversational Systemscited by 28
  6. 2016Learning Executable Semantic Parsers for Natural Language Understandingcited by 21
  7. 2022Python Code Generation by Asking Clarification Questionscited by 16
  8. 2023Zero and Few-shot Semantic Parsing with Ambiguous Inputscited by 16
  9. 2023AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreferencecited by 12
  10. 2023Data Ambiguity Strikes Back: How Documentation Improves GPT's Text-to-SQLcited by 11
  11. 2021Database Search Results Disambiguation for Task-Oriented Dialog Systemscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 16; 2026: 12.

32 papers.

Sort:

By area:

  1. 2024PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queriescited by 17
  2. 2025nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoningcited by 17
  3. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  4. 2025MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmarkcited by 13
  5. 2025Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsingcited by 12
  6. 2025AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQLcited by 10
  7. 2025Asking a Language Model for Diverse Responsescited by 9
  8. 2024PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQLcited by 7
  9. 2024MathBridge: A Large Corpus Dataset for Translating Spoken Mathematical Expressions into LaTeX Formulas for Improved Readabilitycited by 4
  10. 2025BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Basescited by 4
  11. 2025Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunitiescited by 4
  12. 2024Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Typescited by 3
20 more
  1. 2025Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysiscited by 3
  2. 2025Language Models Identify Ambiguities and Exploit Loopholescited by 2
  3. 2026LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queriescited by 2
  4. 2026Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Studycited by 2
  5. 2026CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systemscited by 2
  6. 2025SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gapscited by 1
  7. 2026ODUTQA-MDC: A Task for Open-Domain Underspecified Tabular QA with Multi-turn Dialogue-based Clarificationcited by 1
  8. 2026EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinementcited by 1
  9. 2026EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQLcited by 1
  10. 2025Ambiguity in LLMs is a concept missing problemcited by 0
  11. 2025Belief Attribution as Mental Explanation: The Role of Accuracy, Informativity, and Causalitycited by 0
  12. 2025RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Modelscited by 0
  13. 2025Datrics Text2SQL: A Framework for Natural Language to SQL Query Generationcited by 0
  14. 2025Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQLcited by 0
  15. 2026PleaSQLarify: Visual Pragmatic Repair for Natural Language Database Queryingcited by 0
  16. 2026ROSE: An Intent-Centered Evaluation Metric for NL2SQLcited by 0
  17. 2026Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attributioncited by 0
  18. 2026SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probingcited by 0
  19. 2026Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQLcited by 0
  20. 2026From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMscited by 0