mlsanityData to 14 Oct 2026

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

arXiv 2608.29571 ↗First posted 30 Aug 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Flexible adaptation to context and shared pragmatic intuitions contribute to smooth human conversation. Iterated reference games---in which players repeatedly pick out novel referents using language---present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision--language models on their ability to identify the intended meaning of descriptions produced in iterated reference games, varying the provided context in terms of amount, order, and relevance. While humans performed well consistently, the models we evaluated could make use of prior context to interpret humans' referring expressions, but they struggled to build up the relevant context to interpret those expressions effectively. Our results suggest that the models we evaluated lack core skills needed for efficient linguistic collaboration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 52 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

25 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Measuring How (Not Just Whether) VLMs Build Common Ground4 Sep
  2. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation22 Apr

Other references · 23

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  3. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  4. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  5. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  6. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  7. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
  8. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  9. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  10. 2019Continual adaptation for efficient machine communicationcited by 28
  11. 2021From partners to populations: A hierarchical Bayesian account of coordination and conventioncited by 27
  12. 2020Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contextscited by 22
11 more
  1. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  2. 2022Learning to Mediate Disparities Towards Pragmatic Communicationcited by 14
  3. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMscited by 10
  4. 2025LVLMs are Bad at Overhearing Human Referential Communicationcited by 8
  5. 2026LVLMs and Humans Ground Differently in Referential Communicationcited by 8
  6. 2026MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Gamescited by 8
  7. 2023Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mindcited by 7
  8. 2024CoGen: Learning from Feedback with Coupled Comprehension and Generationcited by 7
  9. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  10. 2026LLMs and people both learn to form conventions -- just not with each othercited by 5
  11. 2025SceneGram: Conceptualizing and Describing Tangrams in Scene Contextcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Context informs pragmatic interpretation in vision-language models
  2. 2026Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests
  3. 2025On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
  4. 2025The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models
  5. 2026Computational models of pragmatic reasoning with flexible generation of meaning and expression alternatives
  6. 2025Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs
  7. 2026A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data
  8. 2026Investigating the Development of Task-Oriented Communication in Vision-Language Models