# ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)

arXiv: https://arxiv.org/abs/2009.11352 · HTML: https://mlsanity.com/paper/2009.11352/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2020.

## Built on by (most cited first, 30 of 58)

- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [GODEL: Large-Scale Pre-Training for Goal-Directed Dialog](https://mlsanity.com/paper/2206.11309/) (2022)
- [Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions](https://mlsanity.com/paper/2205.00415/) (2022)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback](https://mlsanity.com/paper/2304.10750/) (2023)
- [HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?](https://mlsanity.com/paper/2604.09408/) (2026)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [Evaluating Mixed-initiative Conversational Search Systems via User Simulation](https://mlsanity.com/paper/2204.08046/) (2022)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA](https://mlsanity.com/paper/2305.01812/) (2023)
- [STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents](https://mlsanity.com/paper/2405.12059/) (2024)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022)
- [Zero-shot Clarifying Question Generation for Conversational Search](https://mlsanity.com/paper/2301.12660/) (2023)
- [Analysing Mixed Initiatives and Search Strategies during Conversational Search](https://mlsanity.com/paper/2109.05955/) (2021)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models](https://mlsanity.com/paper/2512.21120/) (2025)
- [Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants](https://mlsanity.com/paper/2101.03394/) (2021)
- [Diverse and Specific Clarification Question Generation with Keywords](https://mlsanity.com/paper/2104.10317/) (2021)
- [Collecting Interactive Multi-modal Datasets for Grounded Language Understanding](https://mlsanity.com/paper/2211.06552/) (2022)
- [AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment](https://mlsanity.com/paper/2506.04089/) (2025)
- [Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search](https://mlsanity.com/paper/2402.07742/) (2024)
- [CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models](https://mlsanity.com/paper/2402.06360/) (2024)
- [AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs](https://mlsanity.com/paper/2410.19692/) (2024)
- [Learning to Ask: Conversational Product Search via Representation Learning](https://mlsanity.com/paper/2411.14466/) (2024)
- [Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey](https://mlsanity.com/paper/2505.12543/) (2025)
- [LHAW: Controllable Underspecification for Long-Horizon Tasks](https://mlsanity.com/paper/2602.10525/) (2026)

## Abstract (arXiv)

This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ). The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020. The main aim of the conversational systems is to return an appropriate answer in response to the user requests. However, some user requests might be ambiguous. In IR settings such a situation is handled mainly thought the diversification of the search result page. It is however much more challenging in dialogue settings with limited bandwidth. Therefore, in this challenge, we provide a common evaluation framework to evaluate mixed-initiative conversations. Participants are asked to rank clarifying questions in an information-seeking conversations. The challenge is organized in two stages where in Stage 1 we evaluate the submissions in an offline setting and single-turn conversations. Top participants of Stage 1 get the chance to have their model tested by human annotators.