mlsanityData to 14 Oct 2026

Context informs pragmatic interpretation in vision-language models

arXiv 2511.03908 ↗First posted 5 Nov 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Iterated reference games - in which players repeatedly pick out novel referents using language - present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision-language models on trials from iterated reference games, varying the given context in terms of amount, order, and relevance. Without relevant context, models were above chance but substantially worse than humans. However, with relevant context, model performance increased dramatically over trials. Few-shot reference games with abstract referents remain a difficult task for machine learning models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 27 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

10 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Vision Language Models are Biased29 May
  2. 2025Kimi-VL Technical Report10 Apr
  3. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul

Other references · 7

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2025Gemma 3 Technical Reportcited by 1,824
  3. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  4. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  5. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  6. 2021From partners to populations: A hierarchical Bayesian account of coordination and conventioncited by 27
  7. 2025LVLMs are Bad at Overhearing Human Referential Communicationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
  2. 2026Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests
  3. 2026Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
  4. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation
  5. 2026A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data
  6. 2025Can Vision-Language Models Infer Speaker's Ignorance? The Role of Visual and Linguistic Cues
  7. 2025Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces
  8. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models