mlsanityData to 7 Oct 2026

Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset

arXiv 2302.14406 ↗First posted 28 Feb 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In visual instruction-following dialogue games, players can engage in repair mechanisms in face of an ambiguous or underspecified instruction that cannot be fully mapped to actions in the world. In this work, we annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game. We show that it contains lexically and semantically diverse iCRs being produced self-motivatedly by players deciding to clarify in order to solve the task successfully. With 8.8k iCRs found in 9.9k dialogues, CoDraw-iCR (v1) is a large spontaneous iCR corpus, making it a valuable resource for data-driven research on clarification in dialogue. We then formalise and provide baseline models for two tasks: Determining when to make an iCR and how to recognise them, in order to investigate to what extent these tasks are learnable from data.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 24 of its 67 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

24 of 67 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Vision-and-Dialog Navigationcited by 246
  5. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  6. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  7. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  8. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  9. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  10. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  11. 2021Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledgecited by 40
  12. 2020SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoningcited by 32
12 more
  1. 2020ClarQ: A large-scale and diverse dataset for Clarification Question Generationcited by 31
  2. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  3. 2020Text as Neural Operator: Image Manipulation by Text Instructioncited by 26
  4. 2021NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environmentcited by 15
  5. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  6. 2021A recipe for annotating grounded clarificationscited by 13
  7. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  8. 2021How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpuscited by 11
  9. 2022Collecting Interactive Multi-modal Datasets for Grounded Language Understandingcited by 10
  10. 2021LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulationcited by 5
  11. 2020Interactive Question Clarification in Dialogue via Reinforcement Learningcited by 4
  12. 2022IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 2; 2025: 3; 2026: 1.

11 papers.

  1. 2023Grounding Gaps in Language Model Generationscited by 45
  2. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  3. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 27
  4. 2023Multitask Multimodal Prompted Training for Interactive Embodied Task Completioncited by 9
  5. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  6. 2025Referential ambiguity and clarification requests: comparing human and LLM behaviourcited by 5
  7. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  8. 2023"Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Datasetcited by 3
  9. 2023'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchangescited by 3
  10. 2023Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialoguecited by 1
  11. 2026The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialoguecited by 0