# Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Models

arXiv: https://arxiv.org/abs/2609.18345 · HTML: https://mlsanity.com/paper/2609.18345/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 6 of 6)

- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs](https://mlsanity.com/paper/2406.17126/) (2024)
- [Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals](https://mlsanity.com/paper/2405.20152/) (2024)
- [Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users](https://mlsanity.com/paper/2503.22610/) (2025)
- [@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology](https://mlsanity.com/paper/2409.14215/) (2024)
- [Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models](https://mlsanity.com/paper/2604.27553/) (2026)

## Abstract (arXiv)

Large vision-language models (LVLMs) are commonly used with only a single text prompt as the input, or plus an image. In this paper, we demonstrate that when the image exists, even if the text prompt is not about the specific instance (but only the concept it belongs to) in that image, the response would still be affected. For example, when the text prompt only asks for the attribute descriptions of a dog breed, an image depicting a specific dog from that breed would shift the response. Further, how the specific instance is framed in that image would determine towards which the response shifts. Detailed analyses also reveal that in the response, physical terms increase from 18% for text-only to 45% (40%) for subject-focused (subject-in-situation) framings. Overall, the unexpected effects of visual cues on LVLMs highlight the need to understand the presence of an image and its framing when evaluating the robustness of LVLMs.