# LVLMs and Humans Ground Differently in Referential Communication

arXiv: https://arxiv.org/abs/2601.19792 · HTML: https://mlsanity.com/paper/2601.19792/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 66 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://mlsanity.com/paper/1906.01530/) (2019)
- [Navigating Rifts in Human-LLM Grounding: Study and Benchmark](https://mlsanity.com/paper/2503.13975/) (2025)
- [Continual adaptation for efficient machine communication](https://mlsanity.com/paper/1911.09896/) (2019)
- [Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs](https://mlsanity.com/paper/2408.01417/) (2024)
- [Grounding Language in Multi-Perspective Referential Communication](https://mlsanity.com/paper/2410.03959/) (2024)
- [Measuring How (Not Just Whether) VLMs Build Common Ground](https://mlsanity.com/paper/2509.03805/) (2025)
- [LVLMs are Bad at Overhearing Human Referential Communication](https://mlsanity.com/paper/2509.11514/) (2025)
- [Post-training for Efficient Communication via Convention Formation](https://mlsanity.com/paper/2508.06482/) (2025)
- [Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs](https://mlsanity.com/paper/2503.12370/) (2025)

## Built on by (most cited first, 8 of 8)

- [LLMs and people both learn to form conventions -- just not with each other](https://mlsanity.com/paper/2602.08208/) (2026)
- [Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue](https://mlsanity.com/paper/2606.31719/) (2026)
- [Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventions](https://mlsanity.com/paper/2606.08081/) (2026)
- [Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communication](https://mlsanity.com/paper/2606.17372/) (2026)
- [A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data](https://mlsanity.com/paper/2608.08663/) (2026)
- [When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs](https://mlsanity.com/paper/2608.23978/) (2026)
- [Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation](https://mlsanity.com/paper/2608.29571/) (2026)
- [TRACE: Transparent Retrieval for Abstract Concept Evaluation](https://mlsanity.com/paper/2609.26168/) (2026)

## Abstract (arXiv)

For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. We present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We show that LVLMs cannot interactively generate and resolve referring expressions in a way that enables smooth communication, a crucial skill that underlies human language use. We release our corpus of 356 dialogues (89 pairs over 4 rounds each) along with the online pipeline for data collection and the tools for analyzing accuracy, efficiency, and lexical overlap.