mlsanityData to 14 Oct 2026

Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Models

arXiv 2609.18345 ↗First posted 16 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (LVLMs) are commonly used with only a single text prompt as the input, or plus an image. In this paper, we demonstrate that when the image exists, even if the text prompt is not about the specific instance (but only the concept it belongs to) in that image, the response would still be affected. For example, when the text prompt only asks for the attribute descriptions of a dog breed, an image depicting a specific dog from that breed would shift the response. Further, how the specific instance is framed in that image would determine towards which the response shifts. Detailed analyses also reveal that in the response, physical terms increase from 18% for text-only to 45% (40%) for subject-focused (subject-in-situation) framings. Overall, the unexpected effects of visual cues on LVLMs highlight the need to understand the presence of an image and its framing when evaluating the robustness of LVLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 21 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

6 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models30 Apr
  2. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users28 Mar
  3. 2024@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology21 Sep

Other references · 3

  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  2. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  3. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactualscited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  2. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  3. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions
  4. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
  5. 2026LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
  6. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  7. 2026Seeing Red, Thinking Bad: Color Bias in Vision Language Models
  8. 2024Why do LLaVA Vision-Language Models Reply to Images in English?