# "Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Dataset

arXiv: https://arxiv.org/abs/2306.02377 · HTML: https://mlsanity.com/paper/2306.02377/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 27 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [A recipe for annotating grounded clarifications](https://mlsanity.com/paper/2104.08964/) (2021)
- [How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpus](https://mlsanity.com/paper/2106.06504/) (2021)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)

## Built on by (most cited first, 3 of 3)

- [Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests](https://mlsanity.com/paper/2401.17039/) (2024)
- [Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling](https://mlsanity.com/paper/2306.12387/) (2023)
- [Improving Agent Interactions in Virtual Environments with Language Models](https://mlsanity.com/paper/2402.05440/) (2024)

## Abstract (arXiv)

Instruction Clarification Requests are a mechanism to solve communication problems, which is very functional in instruction-following interactions. Recent work has argued that the CoDraw dataset is a valuable source of naturally occurring iCRs. Beyond identifying when iCRs should be made, dialogue models should also be able to generate them with suitable form and content. In this work, we introduce CoDraw-iCR (v2), extending the existing iCR identifiers with fine-grained information grounded in the underlying dialogue game items and possible actions. Our annotation can serve to model and evaluate repair capabilities of dialogue agents.