# ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog

arXiv: https://arxiv.org/abs/2409.06097 · HTML: https://mlsanity.com/paper/2409.06097/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2024.

## Built on by (most cited first, 24 of 24)

- [Creating General User Models from Computer Use](https://mlsanity.com/paper/2505.10831/) (2025)
- [Navigating Rifts in Human-LLM Grounding: Study and Benchmark](https://mlsanity.com/paper/2503.13975/) (2025)
- [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026)
- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models](https://mlsanity.com/paper/2512.21120/) (2025)
- [Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments](https://mlsanity.com/paper/2605.27209/) (2026)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [ProactBench: Beyond What The User Asked For](https://mlsanity.com/paper/2605.09228/) (2026)
- [Agentic Coding Needs Proactivity, Not Just Autonomy](https://mlsanity.com/paper/2605.06717/) (2026)
- [Pause or Fabricate? Training Language Models for Grounded Reasoning](https://mlsanity.com/paper/2604.19656/) (2026)
- [SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science](https://mlsanity.com/paper/2605.18630/) (2026)
- [Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases](https://mlsanity.com/paper/2502.14507/) (2025)
- [Program Synthesis Dialog Agents for Interactive Decision-Making](https://mlsanity.com/paper/2502.19610/) (2025)
- [MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language](https://mlsanity.com/paper/2505.14395/) (2025)
- [One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies](https://mlsanity.com/paper/2607.21143/) (2026)
- [InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries](https://mlsanity.com/paper/2608.20220/) (2026)
- [Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent](https://mlsanity.com/paper/2609.38604/) (2026)
- [RPS: Information Elicitation with Reinforcement Prompt Selection](https://mlsanity.com/paper/2604.13817/) (2026)
- [CA-BED: Conversation-Aware Bayesian Experimental Design](https://mlsanity.com/paper/2606.01182/) (2026)
- [MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning](https://mlsanity.com/paper/2607.07391/) (2026)
- [LOGOS: A Living Logic for AI Agent Teams That Evolve With Humans](https://mlsanity.com/paper/2607.10878/) (2026)
- [CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement](https://mlsanity.com/paper/2608.11631/) (2026)
- [A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models](https://mlsanity.com/paper/2609.02054/) (2026)
- [CLEAT: Learning Value-Guided Clarification for Interactive Agents](https://mlsanity.com/paper/2610.12655/) (2026)

## Abstract (arXiv)

We introduce ClarQ-LLM, an evaluation framework consisting of bilingual English-Chinese conversation tasks, conversational agents and evaluation metrics, designed to serve as a strong benchmark for assessing agents' ability to ask clarification questions in task-oriented dialogues. The benchmark includes 31 different task types, each with 10 unique dialogue scenarios between information seeker and provider agents. The scenarios require the seeker to ask questions to resolve uncertainty and gather necessary information to complete tasks. Unlike traditional benchmarks that evaluate agents based on fixed dialogue content, ClarQ-LLM includes a provider conversational agent to replicate the original human provider in the benchmark. This allows both current and future seeker agents to test their ability to complete information gathering tasks through dialogue by directly interacting with our provider agent. In tests, LLAMA3.1 405B seeker agent managed a maximum success rate of only 60.05%, showing that ClarQ-LLM presents a strong challenge for future research.