mlsanityData to 7 Oct 2026

LVLMs and Humans Ground Differently in Referential Communication

arXiv 2601.19792 ↗First posted 27 Jan 2026cs.CL cs.AI cs.HCcited by 8

Not on the map. Why?

Abstract (arXiv)

For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. We present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We show that LVLMs cannot interactively generate and resolve referring expressions in a way that enables smooth communication, a crucial skill that underlies human language use. We release our corpus of 356 dialogues (89 pairs over 4 rounds each) along with the online pipeline for data collection and the tools for analyzing accuracy, efficiency, and lexical overlap.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 11 of its 66 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

11 of 66 references.

References · 11

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2019The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialoguecited by 46
  4. 2019Continual adaptation for efficient machine communicationcited by 28
  5. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 27
  6. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  7. 2024Grounding Language in Multi-Perspective Referential Communicationcited by 12
  8. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  9. 2025LVLMs are Bad at Overhearing Human Referential Communicationcited by 8
  10. 2025Post-training for Efficient Communication via Convention Formationcited by 7
  11. 2025Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026LLMs and people both learn to form conventions -- just not with each othercited by 5
  2. 2026Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialoguecited by 1
  3. 2026Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventionscited by 0
  4. 2026Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communicationcited by 0
  5. 2026A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Datacited by 0
  6. 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMscited by 0
  7. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0
  8. 2026TRACE: Transparent Retrieval for Abstract Concept Evaluationcited by 0