# Can Vision-Language Models Infer Speaker's Ignorance? The Role of Visual and Linguistic Cues

arXiv: https://arxiv.org/abs/2502.09120 · HTML: https://mlsanity.com/paper/2502.09120/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 54 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Context-Aware Crowd Counting](https://mlsanity.com/paper/1811.10452/) (2018)
- [Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts](https://mlsanity.com/paper/1904.09073/) (2019)
- [Evaluating statistical language models as pragmatic reasoners](https://mlsanity.com/paper/2305.01020/) (2023)
- [Expectations over Unspoken Alternatives Predict Pragmatic Inferences](https://mlsanity.com/paper/2304.04758/) (2023)
- [Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions](https://mlsanity.com/paper/2405.05776/) (2024)
- [Pragmatic inference of scalar implicature by LLMs](https://mlsanity.com/paper/2408.06673/) (2024)
- [Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks](https://mlsanity.com/paper/1905.07075/) (2019)

## Abstract (arXiv)

This study investigates whether vision-language models (VLMs) can perform pragmatic inference, focusing on ignorance implicatures, utterances that imply the speaker's lack of precise knowledge. To test this, we systematically manipulated contextual cues: the visually depicted situation (visual cue) and QUD-based linguistic prompts (linguistic cue). When only visual cues were provided, three state-of-the-art VLMs (GPT-4o, Gemini 1.5 Pro, and Claude 3.5 sonnet) produced interpretations largely based on the lexical meaning of the modified numerals. When linguistic cues were added to enhance contextual informativeness, Claude exhibited more human-like inference by integrating both types of contextual cues. In contrast, GPT and Gemini favored precise, literal interpretations. Although the influence of contextual cues increased, they treated each contextual cue independently and aligned them with semantic features rather than engaging in context-driven reasoning. These findings suggest that although the models differ in how they handle contextual cues, Claude's ability to combine multiple cues may signal emerging pragmatic competence in multimodal models.