# Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift

arXiv: https://arxiv.org/abs/2608.29543 · HTML: https://mlsanity.com/paper/2608.29543/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?](https://mlsanity.com/paper/2604.09408/) (2026)
- [Agentar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scaling](https://mlsanity.com/paper/2509.24403/) (2025)
- [From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems](https://mlsanity.com/paper/2410.01066/) (2024)
- [Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration](https://mlsanity.com/paper/2510.26495/) (2025)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.