# RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs

arXiv: https://arxiv.org/abs/2412.13835 · HTML: https://mlsanity.com/paper/2412.13835/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 66 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Why Does a Visual Question Have Different Answers?](https://mlsanity.com/paper/1908.04342/) (2019)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA](https://mlsanity.com/paper/2211.07516/) (2022)
- [Do You See What I Mean? Visual Resolution of Linguistic Ambiguities](https://mlsanity.com/paper/1603.08079/) (2016)
- [Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!](https://mlsanity.com/paper/2402.12486/) (2024)
- ['What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges](https://mlsanity.com/paper/2307.15554/) (2023)

## Built on by (most cited first, 8 of 8)

- [It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledge](https://mlsanity.com/paper/2509.16107/) (2025)
- [Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests](https://mlsanity.com/paper/2601.07820/) (2026)
- [Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA](https://mlsanity.com/paper/2604.17316/) (2026)
- [IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models](https://mlsanity.com/paper/2602.16138/) (2026)
- [Clarify, Abstain or Answer? Strategising in Conversation with Belief-Augmented Generation](https://mlsanity.com/paper/2605.25831/) (2026)
- [Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act](https://mlsanity.com/paper/2609.22588/) (2026)
- [Using Perspectival Words Is Harder Than Vocabulary Words for Humans and Even More So for Multimodal Language Models](https://mlsanity.com/paper/2506.00065/) (2025)
- [Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs](https://mlsanity.com/paper/2506.00258/) (2025)

## Abstract (arXiv)

Ambiguity resolution is key to effective communication. While humans effortlessly address ambiguity through conversational grounding strategies, the extent to which current language models can emulate these strategies remains unclear. In this work, we examine referential ambiguity in image-based question answering by introducing RACQUET, a carefully curated dataset targeting distinct aspects of ambiguity. Through a series of evaluations, we reveal significant limitations and problems of overconfidence of state-of-the-art large multimodal language models in addressing ambiguity in their responses. The overconfidence issue becomes particularly relevant for RACQUET-BIAS, a subset designed to analyze a critical yet underexplored problem: failing to address ambiguity leads to stereotypical, socially biased responses. Our results underscore the urgency of equipping models with robust strategies to deal with uncertainty without resorting to undesirable stereotypes.