CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems
Not on the map. Why?
Abstract (arXiv)
NL2SQL systems deployed in industry settings often encounter ambiguous or unanswerable queries, particularly in interactive scenarios with incomplete user clarification. Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution, overlooking realistic failure modes. We introduce Clarity, a framework for automatically generating an NL2SQL benchmark with multi-faceted ambiguities and diverse user behaviors across both single- and multi-turn settings. Using a constraint-driven pipeline, Clarity transforms executable SQL into ambiguous queries, augmented with grounded conversational continuations and schema-level metadata. Empirical evaluation on Spider and BIRD shows that leading NL2SQL systems, including those based on strong LLMs, suffer significant performance degradation under multi-faceted ambiguity. While these systems often detect ambiguity, they struggle to accurately localize and resolve the underlying schema-level sources. Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2SQL systems.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 24 Apr 2026.
Builds on
21 of 30 references.
References · 21
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
- 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
- 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
- 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 103
- 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
- 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
- 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 32
- 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 31
- 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 19
9 more
- 2024PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queriescited by 17
- 2025nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoningcited by 17
- 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
- 2024SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Mergingcited by 16
- 2025SQLForge: Synthesizing Reliable and Diverse Data to Enhance Text-to-SQL Reasoning in LLMscited by 10
- 2025AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQLcited by 10
- 2025SING-SQL: A Synthetic Data Generation Framework for In-Domain Text-to-SQL Translationcited by 5
- 2024Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Typescited by 3
- 2025ODIN: A NL2SQL Recommender to Handle Schema Ambiguitycited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.