# CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

arXiv: https://arxiv.org/abs/2604.22313 · HTML: https://mlsanity.com/paper/2604.22313/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation](https://mlsanity.com/paper/2405.07467/) (2024)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries](https://mlsanity.com/paper/2406.19073/) (2024)
- [Benchmarking and Improving Text-to-SQL Generation under Ambiguity](https://mlsanity.com/paper/2310.13659/) (2023)
- [Reliable Text-to-SQL with Adaptive Abstention](https://mlsanity.com/paper/2501.10858/) (2025)
- [PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries](https://mlsanity.com/paper/2410.11076/) (2024)
- [nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoning](https://mlsanity.com/paper/2503.12880/) (2025)
- [BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions](https://mlsanity.com/paper/2510.05318/) (2025)
- [SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging](https://mlsanity.com/paper/2408.12733/) (2024)
- [SQLForge: Synthesizing Reliable and Diverse Data to Enhance Text-to-SQL Reasoning in LLMs](https://mlsanity.com/paper/2505.13725/) (2025)
- [AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL](https://mlsanity.com/paper/2508.15276/) (2025)
- [SING-SQL: A Synthetic Data Generation Framework for In-Domain Text-to-SQL Translation](https://mlsanity.com/paper/2509.25672/) (2025)
- [ODIN: A NL2SQL Recommender to Handle Schema Ambiguity](https://mlsanity.com/paper/2505.19302/) (2025)
- [Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types](https://mlsanity.com/paper/2412.17867/) (2024)

## Built on by (most cited first, 2 of 2)

- [Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization](https://mlsanity.com/paper/2609.05258/) (2026)
- [Clarification Is Not Correction: LLMs Fail to Let Go](https://mlsanity.com/paper/2609.25337/) (2026)

## Abstract (arXiv)

NL2SQL systems deployed in industry settings often encounter ambiguous or unanswerable queries, particularly in interactive scenarios with incomplete user clarification. Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution, overlooking realistic failure modes. We introduce Clarity, a framework for automatically generating an NL2SQL benchmark with multi-faceted ambiguities and diverse user behaviors across both single- and multi-turn settings. Using a constraint-driven pipeline, Clarity transforms executable SQL into ambiguous queries, augmented with grounded conversational continuations and schema-level metadata. Empirical evaluation on Spider and BIRD shows that leading NL2SQL systems, including those based on strong LLMs, suffer significant performance degradation under multi-faceted ambiguity. While these systems often detect ambiguity, they struggle to accurately localize and resolve the underlying schema-level sources. Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2SQL systems.