Can Vision-Language Models Infer Speaker's Ignorance? The Role of Visual and Linguistic Cues
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
This study investigates whether vision-language models (VLMs) can perform pragmatic inference, focusing on ignorance implicatures, utterances that imply the speaker's lack of precise knowledge. To test this, we systematically manipulated contextual cues: the visually depicted situation (visual cue) and QUD-based linguistic prompts (linguistic cue). When only visual cues were provided, three state-of-the-art VLMs (GPT-4o, Gemini 1.5 Pro, and Claude 3.5 sonnet) produced interpretations largely based on the lexical meaning of the modified numerals. When linguistic cues were added to enhance contextual informativeness, Claude exhibited more human-like inference by integrating both types of contextual cues. In contrast, GPT and Gemini favored precise, literal interpretations. Although the influence of contextual cues increased, they treated each contextual cue independently and aligned them with semantic features rather than engaging in context-driven reasoning. These findings suggest that although the models differ in how they handle contextual cues, Claude's ability to combine multiple cues may signal emerging pragmatic competence in multimodal models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 54 references that are in the map. First posted on arXiv 13 Feb 2025.
Builds on
17 of 54 references.
Closest ancestors · 1
Same direction, newest first
- 2023Teaching CLIP to Count to Ten23 Feb
Other references · 16
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2015VQA: Visual Question Answeringcited by 2,489
- 2014Show and Tell: A Neural Image Caption Generatorcited by 877
- 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
- 2018Context-Aware Crowd Countingcited by 150
- 2019Integrating Text and Image: Determining Multimodal Document Intent in Instagram Postscited by 37
- 2023Evaluating statistical language models as pragmatic reasonerscited by 14
4 more
- 2023Expectations over Unspoken Alternatives Predict Pragmatic Inferencescited by 9
- 2024Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctionscited by 5
- 2024Pragmatic inference of scalar implicature by LLMscited by 5
- 2019Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networkscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
- 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
- 2025Context informs pragmatic interpretation in vision-language models
- 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
- 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
- 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
- 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
- 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation