mlsanityData to 7 Oct 2026

Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

arXiv 2109.05794 ↗First posted 13 Sep 2021cs.CL cs.AI cs.IRcited by 63

Not on the map. Why?

Abstract (arXiv)

Enabling open-domain dialogue systems to ask clarifying questions when appropriate is an important direction for improving the quality of the system response. Namely, for cases when a user request is not specific enough for a conversation system to provide an answer right away, it is desirable to ask a clarifying question to increase the chances of retrieving a satisfying answer. To address the problem of 'asking clarifying questions in open-domain dialogues': (1) we collect and release a new dataset focused on open-domain single- and multi-turn conversations, (2) we benchmark several state-of-the-art neural baselines, and (3) we propose a pipeline consisting of offline and online steps for evaluating the quality of clarifying questions in various dialogues. These contributions are suitable as a foundation for further research.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 32 of its 67 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

32 of 67 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  4. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  5. 2020Recipes for building an open-domain chatbotcited by 593
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  7. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  8. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  9. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  10. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  11. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  12. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
20 more
  1. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  2. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  3. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  4. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  5. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  6. 2016A User Simulator for Task-Completion Dialoguescited by 56
  7. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 55
  8. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  9. 2019Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialogcited by 37
  10. 2020MIMICS: A Large-Scale Data Collection for Search Clarificationcited by 27
  11. 2018Learning to Ask Questions in Open-domain Conversational Systems with Typed Decoderscited by 26
  12. 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learningcited by 21
  13. 2019Interconnected Question Generation with Coreference Alignment and Conversation Flow Modelingcited by 20
  14. 2020Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Searchcited by 13
  15. 2019Identifying Unclear Questions in Community Question Answering Websitescited by 12
  16. 2021User Engagement Prediction for Clarification in Searchcited by 12
  17. 2018Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learningcited by 11
  18. 2021DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialoguescited by 7
  19. 2021Ranking Clarifying Questions Based on Predicted User Engagementcited by 6
  20. 2020A Clarifying Question Selection System from NTES_ALONG in Convai3 Challengecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 8; 2024: 22; 2025: 9; 2026: 12.

63 papers.

Sort:

By area:

  1. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  3. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  4. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  5. 2023A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospectscited by 66
  6. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 59
  7. 2024Towards Human-centered Proactive Conversational Agentscited by 48
  8. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 48
  9. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  10. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  11. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  12. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
38 more
  1. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  2. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 31
  3. 2022INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactionscited by 27
  4. 2022Task Ambiguity in Humans and Language Modelscited by 27
  5. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  6. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 24
  7. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  8. 2024A Survey of Conversational Searchcited by 22
  9. 2023Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QAcited by 20
  10. 2024STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agentscited by 17
  11. 2022Python Code Generation by Asking Clarification Questionscited by 16
  12. 2021NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environmentcited by 15
  13. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  14. 2023Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructionscited by 13
  15. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  16. 2022Collecting Interactive Multi-modal Datasets for Grounded Language Understandingcited by 10
  17. 2024ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questionscited by 10
  18. 2021Database Search Results Disambiguation for Task-Oriented Dialog Systemscited by 9
  19. 2024Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Searchcited by 8
  20. 2025Conversational Search: From Fundamentals to Frontiers in the LLM Eracited by 8
  21. 2023Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialoguescited by 7
  22. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  23. 2026MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likenesscited by 7
  24. 2025Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Surveycited by 6
  25. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5
  26. 2024Estimating the Usefulness of Clarifying Questions and Answers for Conversational Searchcited by 4
  27. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  28. 2024Redefining Proactivity for Information Seeking Dialoguecited by 4
  29. 2025Zero-Shot and Efficient Clarification Need Prediction in Conversational Searchcited by 4
  30. 2025Query Understanding in LLM-based Conversational Information Seekingcited by 4
  31. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  32. 2023'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchangescited by 3
  33. 2024Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Modelscited by 3
  34. 2025Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understandingcited by 3
  35. 2025Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approachcited by 3
  36. 2026IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agentscited by 3
  37. 2022PREME: Preference-based Meeting Exploration through an Interactive Questionnairecited by 2
  38. 2022CLSE: Corpus of Linguistically Significant Entitiescited by 2

50 most cited of 63 All 63 (JSON)