mlsanityData to 14 Oct 2026

LVLMs are Bad at Overhearing Human Referential Communication

arXiv 2509.11514 ↗First posted 15 Sep 2025cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

During spontaneous conversations, speakers collaborate on novel referring expressions, which they can then re-use in subsequent conversations. Understanding such referring expressions is an important ability for an embodied agent, so that it can carry out tasks in the real world. This requires integrating and understanding language, vision, and conversational interaction. We study the capabilities of seven state-of-the-art Large Vision Language Models (LVLMs) as overhearers to a corpus of spontaneous conversations between pairs of human discourse participants engaged in a collaborative object-matching task. We find that such a task remains challenging for current LVLMs and they all fail to show a consistent performance improvement as they overhear more conversations from the same discourse participants repeating the same task for multiple rounds. We release our corpus and code for reproducibility and to facilitate future research.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 17 of its 50 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

17 of 50 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  6. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  7. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 222
  8. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  9. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  10. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  11. 2024LLMs achieve adult human performance on higher-order theory of mind taskscited by 61
  12. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
5 more
  1. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  2. 2024ScanFormer: Referring Expression Comprehension by Iteratively Scanningcited by 18
  3. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  4. 2024Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Groundcited by 10
  5. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Typescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026LVLMs and Humans Ground Differently in Referential Communicationcited by 8
  2. 2025Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTaskcited by 2
  3. 2026Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialoguecited by 1
  4. 2025Context informs pragmatic interpretation in vision-language modelscited by 0
  5. 2026Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventionscited by 0
  6. 2026Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communicationcited by 0
  7. 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMscited by 0
  8. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation
  2. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
  3. 2026A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data
  4. 2025Towards Multimodal Social Conversations with Robots: Using Vision-Language Models
  5. 2026Learning to Refer from Estimated Listener Gaze
  6. 2025Measuring How (Not Just Whether) VLMs Build Common Ground
  7. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models
  8. 2025Analyzing the Sensitivity of Vision Language Models in Visual Question Answering