# Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

arXiv: https://arxiv.org/abs/2601.07820 · HTML: https://mlsanity.com/paper/2601.07820/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding](https://mlsanity.com/paper/1703.10186/) (2017)
- [Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong](https://mlsanity.com/paper/2310.12558/) (2023)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [A Diachronic Perspective on User Trust in AI under Uncertainty](https://mlsanity.com/paper/2310.13544/) (2023)
- [LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High](https://mlsanity.com/paper/2505.22354/) (2025)
- [Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis](https://mlsanity.com/paper/2309.15656/) (2023)
- [Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions](https://mlsanity.com/paper/2506.08952/) (2025)
- [RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs](https://mlsanity.com/paper/2412.13835/) (2024)
- [Referential ambiguity and clarification requests: comparing human and LLM behaviour](https://mlsanity.com/paper/2507.10445/) (2025)
- [Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks?](https://mlsanity.com/paper/2506.11807/) (2025)

## Built on by (most cited first, 2 of 2)

- [When Contextual Inference Fails: Cancelability in Interactive Instruction Following](https://mlsanity.com/paper/2603.19997/) (2026)
- [Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models](https://mlsanity.com/paper/2609.20846/) (2026)

## Abstract (arXiv)

In human conversation, both interlocutors play an active role in maintaining mutual understanding. When listeners are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar listener role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a suitable testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.