# A recipe for annotating grounded clarifications

arXiv: https://arxiv.org/abs/2104.08964 · HTML: https://mlsanity.com/paper/2104.08964/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 55 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)

## Built on by (most cited first, 13 of 13)

- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment](https://mlsanity.com/paper/2506.04089/) (2025)
- [Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests](https://mlsanity.com/paper/2401.17039/) (2024)
- ["Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Dataset](https://mlsanity.com/paper/2306.02377/) (2023)
- ['What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges](https://mlsanity.com/paper/2307.15554/) (2023)
- [Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models](https://mlsanity.com/paper/2409.14247/) (2024)
- [Discourse over Discourse: The Need for an Expanded Pragmatic Focus in Conversational AI](https://mlsanity.com/paper/2304.14543/) (2023)
- [Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2309.10375/) (2023)
- [It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning](https://mlsanity.com/paper/2405.01139/) (2024)
- [Expanding the Set of Pragmatic Considerations in Conversational AI](https://mlsanity.com/paper/2310.18435/) (2023)

## Abstract (arXiv)

In order to interpret the communicative intents of an utterance, it needs to be grounded in something that is outside of language; that is, grounded in world modalities. In this paper, we argue that dialogue clarification mechanisms make explicit the process of interpreting the communicative intents of the speaker's utterances by grounding them in the various modalities in which the dialogue is situated. This paper frames dialogue clarification mechanisms as an understudied research problem and a key missing piece in the giant jigsaw puzzle of natural language understanding. We discuss both the theoretical background and practical challenges posed by this problem and propose a recipe for obtaining grounding annotations. We conclude by highlighting ethical issues that need to be addressed in future work.