# Measuring How (Not Just Whether) VLMs Build Common Ground

arXiv: https://arxiv.org/abs/2509.03805 · HTML: https://mlsanity.com/paper/2509.03805/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Why Does the Effective Context Length of LLMs Fall Short?](https://mlsanity.com/paper/2410.18745/) (2024)
- [Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings](https://mlsanity.com/paper/1704.07130/) (2017)
- [Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding](https://mlsanity.com/paper/1703.10186/) (2017)
- [Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents](https://mlsanity.com/paper/2305.13455/) (2023)
- [The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://mlsanity.com/paper/1906.01530/) (2019)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)
- [Continual adaptation for efficient machine communication](https://mlsanity.com/paper/1911.09896/) (2019)
- [Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts](https://mlsanity.com/paper/2011.04554/) (2020)
- [Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs](https://mlsanity.com/paper/2408.01417/) (2024)
- [LEATHER: A Framework for Learning to Generate Human-like Text in Dialogue](https://mlsanity.com/paper/2210.07777/) (2022)

## Built on by (most cited first, 8 of 8)

- [LVLMs and Humans Ground Differently in Referential Communication](https://mlsanity.com/paper/2601.19792/) (2026)
- [Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks](https://mlsanity.com/paper/2607.29585/) (2026)
- [Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry](https://mlsanity.com/paper/2603.05450/) (2026)
- [Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventions](https://mlsanity.com/paper/2606.08081/) (2026)
- [A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data](https://mlsanity.com/paper/2608.08663/) (2026)
- [When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs](https://mlsanity.com/paper/2608.23978/) (2026)
- [Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation](https://mlsanity.com/paper/2608.29571/) (2026)
- [TRACE: Transparent Retrieval for Abstract Concept Evaluation](https://mlsanity.com/paper/2609.26168/) (2026)

## Abstract (arXiv)

Large vision language models (VLMs) increasingly claim reasoning skills, yet current benchmarks evaluate them in single-turn or question answering settings. However, grounding is an interactive process in which people gradually develop shared understanding through ongoing communication. We introduce a four-metric suite (grounding efficiency, content alignment, lexical adaptation, and human-likeness) to systematically evaluate VLM performance in interactive grounding contexts. We deploy the suite on 150 self-play sessions of interactive referential games between three proprietary VLMs and compare them with human dyads. All three models diverge from human patterns on at least three metrics, while GPT4o-mini is the closest overall. We find that (i) task success scores do not indicate successful grounding and (ii) high image-utterance alignment does not necessarily predict task success. Our metric suite and findings offer a framework for future research on VLM grounding.