mlsanityData to 14 Oct 2026

Measuring How (Not Just Whether) VLMs Build Common Ground

arXiv 2509.03805 ↗First posted 4 Sep 2025cs.CL cs.AIcited by 8

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large vision language models (VLMs) increasingly claim reasoning skills, yet current benchmarks evaluate them in single-turn or question answering settings. However, grounding is an interactive process in which people gradually develop shared understanding through ongoing communication. We introduce a four-metric suite (grounding efficiency, content alignment, lexical adaptation, and human-likeness) to systematically evaluate VLM performance in interactive grounding contexts. We deploy the suite on 150 self-play sessions of interactive referential games between three proprietary VLMs and compare them with human dyads. All three models diverge from human patterns on at least three metrics, while GPT4o-mini is the closest overall. We find that (i) task success scores do not indicate successful grounding and (ii) high image-utterance alignment does not necessarily predict task success. Our metric suite and findings offer a framework for future research on VLM grounding.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 43 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

18 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Abstract Visual Reasoning with Tangram Shapes29 Nov

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  9. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  10. 2017Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddingscited by 59
  11. 2017Colors in Context: A Pragmatic Neural Model for Grounded Language Understandingcited by 57
  12. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 54
5 more
  1. 2019The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialoguecited by 46
  2. 2019Continual adaptation for efficient machine communicationcited by 28
  3. 2020Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contextscited by 22
  4. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  5. 2022LEATHER: A Framework for Learning to Generate Human-like Text in Dialoguecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026LVLMs and Humans Ground Differently in Referential Communicationcited by 8
  2. 2026Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Taskscited by 2
  3. 2026Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetrycited by 1
  4. 2026Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventionscited by 0
  5. 2026A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Datacited by 0
  6. 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMscited by 0
  7. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0
  8. 2026TRACE: Transparent Retrieval for Abstract Concept Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
  2. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?
  3. 2025Towards Understanding Visual Grounding in Visual Language Models
  4. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models
  5. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  6. 2026Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models
  7. 2025The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs
  8. 2026Multi-Turn Multi-Agent Dialogue for Collaborative Reconstruction Improves VLM Performance on Spatial Reasoning, But Only Barely