mlsanityData to 14 Oct 2026

Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery

arXiv 2605.27315 ↗First posted 26 May 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual inputs are often assumed to improve language understanding in multimodal models. We examine this assumption by asking whether vision-language models (VLMs) can distinguish useful visual evidence from incidental image context in lexical judgments. We use human concreteness and imagery ratings because they span words with varying expected visual relevance, from abstract and low-imagery words to concrete and high-imagery words. We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings, most sharply when visual evidence is least relevant. Through probing and canonical correlation analysis, complemented by an attribution case study, we find that real-image contexts are associated with representational shifts and greater sensitivity to spurious visual cues, coinciding with weaker recoverability of the targeted lexical properties. We further show that instructing models to focus solely on textual content at inference time can reduce this degradation, with the clearest gains on these vulnerable subsets. Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 48 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

21 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs27 Jan
  2. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May
  3. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts24 Jun

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  9. 2019Designing and Interpreting Probes with Control Taskscited by 640
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  11. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
6 more
  1. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 72
  2. 2019Visually Grounded Neural Syntax Acquisitioncited by 48
  3. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 20
  4. 2023Visual Grounding Helps Learn Word Meanings in Low-Data Regimescited by 17
  5. 2024Lexicon-Level Contrastive Visual-Grounding Improves Language Modelingcited by 6
  6. 2025Experiential Semantic Information and Brain Alignment: Are Multimodal Models Better than Language Models?cited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  2. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
  3. 2026Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language Models
  4. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
  5. 2026ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
  6. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  7. 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
  8. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models