# ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models

arXiv: https://arxiv.org/abs/2512.21120 · HTML: https://mlsanity.com/paper/2512.21120/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 24 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)](https://mlsanity.com/paper/2009.11352/) (2020)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog](https://mlsanity.com/paper/2409.06097/) (2024)
- [From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps](https://mlsanity.com/paper/2504.02052/) (2025)
- [An In-depth Investigation of User Response Simulation for Conversational Search](https://mlsanity.com/paper/2304.07944/) (2023)
- [Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation](https://mlsanity.com/paper/2504.12113/) (2025)
- [Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding](https://mlsanity.com/paper/2502.11442/) (2025)

## Built on by (most cited first, 13 of 13)

- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction](https://mlsanity.com/paper/2602.02455/) (2026)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models](https://mlsanity.com/paper/2605.30251/) (2026)
- [Pause or Fabricate? Training Language Models for Grounded Reasoning](https://mlsanity.com/paper/2604.19656/) (2026)
- [InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?](https://mlsanity.com/paper/2604.27419/) (2026)
- [Interactive Task Alignment as a POMDP](https://mlsanity.com/paper/2607.16412/) (2026)
- [One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies](https://mlsanity.com/paper/2607.21143/) (2026)
- [InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries](https://mlsanity.com/paper/2608.20220/) (2026)
- [Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents](https://mlsanity.com/paper/2602.07796/) (2026)
- [CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMs](https://mlsanity.com/paper/2608.00018/) (2026)
- [What Happens During Autonomous Deep Research After the User Steps Away?](https://mlsanity.com/paper/2609.33509/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed as conversational assistants in open-domain, multi-turn settings, where users often provide incomplete or ambiguous information. However, existing LLM-focused clarification benchmarks primarily assume single-turn interactions or cooperative users, limiting their ability to evaluate clarification behavior in realistic settings. We introduce ClarifyMT-Bench, a benchmark for multi-turn clarification grounded in a five-dimensional ambiguity taxonomy and a set of six behaviorally diverse simulated user personas. Through a hybrid LLM-human pipeline, we construct 6,120 multi-turn dialogues capturing diverse ambiguity sources and interaction patterns. Evaluating ten representative LLMs uncovers a consistent under-clarification bias: LLMs tend to answer prematurely, and performance degrades as dialogue depth increases. To mitigate this, we propose ClarifyAgent, an agentic approach that decomposes clarification into perception, forecasting, tracking, and planning, substantially improving robustness across ambiguity conditions. ClarifyMT-Bench establishes a reproducible foundation for studying when LLMs should ask, when they should answer, and how to navigate ambiguity in real-world human-LLM interactions.