# Referential ambiguity and clarification requests: comparing human and LLM behaviour

arXiv: https://arxiv.org/abs/2507.10445 · HTML: https://mlsanity.com/paper/2507.10445/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Dialog Acts for Task-Driven Embodied Agents](https://mlsanity.com/paper/2209.12953/) (2022)
- [How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpus](https://mlsanity.com/paper/2106.06504/) (2021)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [Empowering Language Models with Active Inquiry for Deeper Understanding](https://mlsanity.com/paper/2402.03719/) (2024)
- [[Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Literary Coreference Annotation with LLMs](https://mlsanity.com/paper/2401.17922/) (2024)
- [A LLM Benchmark based on the Minecraft Builder Dialog Agent Task](https://mlsanity.com/paper/2407.12734/) (2024)
- [MDC-R: The Minecraft Dialogue Corpus with Reference](https://mlsanity.com/paper/2506.22062/) (2025)

## Built on by (most cited first, 5 of 5)

- [Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks](https://mlsanity.com/paper/2604.14624/) (2026)
- [Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests](https://mlsanity.com/paper/2601.07820/) (2026)
- [Frame of Reference: Addressing the Challenges of Common Ground Representation in Situational Dialogs](https://mlsanity.com/paper/2601.09365/) (2026)
- [Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs](https://mlsanity.com/paper/2604.19245/) (2026)
- [Referential Uncertainty in Human--AI Collaboration](https://mlsanity.com/paper/2609.39518/) (2026)

## Abstract (arXiv)

In this work we examine LLMs' ability to ask clarification questions in task-oriented dialogues that follow the asynchronous instruction-giver/instruction-follower format. We present a new corpus that combines two existing annotations of the Minecraft Dialogue Corpus -- one for reference and ambiguity in reference, and one for SDRT including clarifications -- into a single common format providing the necessary information to experiment with clarifications and their relation to ambiguity. With this corpus we compare LLM actions with original human-generated clarification questions, examining how both humans and LLMs act in the case of ambiguity. We find that there is only a weak link between ambiguity and humans producing clarification questions in these dialogues, and low correlation between humans and LLMs. Humans hardly ever produce clarification questions for referential ambiguity, but often do so for task-based uncertainty. Conversely, LLMs produce more clarification questions for referential ambiguity, but less so for task uncertainty. We question if LLMs' ability to ask clarification questions is predicated on their recent ability to simulate reasoning, and test this with different reasoning approaches, finding that reasoning does appear to increase question frequency and relevancy.