Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Recent research on Vision Language Models (VLMs) suggests that they rely on inherent biases learned during training to respond to questions about visual properties of an image. These biases are exacerbated when VLMs are asked highly specific questions that require focusing on specific areas of the image. For example, a VLM tasked with counting stars on a modified American flag (e.g., with more than 50 stars) will often disregard the visual evidence and fail to answer accurately. We build upon this research and develop a multi-dimensional examination framework to systematically determine which characteristics of the input data, including both the image and the accompanying prompt, lead to such differences in performance. Using open-source VLMs, we further examine how attention values fluctuate with varying input parameters (e.g., image size, number of objects in the image, background color, prompt specificity). This research aims to learn how the behavior of vision language models changes and to explore methods for characterizing such changes. Our results suggest, among other things, that even minor modifications in image characteristics and prompt specificity can lead to large changes in how a VLM formulates its answer and, subsequently, its overall performance.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 11 references that are in the map. First posted on arXiv 10 Sep 2025.
Builds on
7 of 11 references.
Closest ancestors · 3
Same direction, newest first
- 2024VHELM: A Holistic Evaluation of Vision Language Models9 Oct
- 2024Do Vision-Language Models Really Understand Visual Language?30 Sep
- 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models13 Jun
Other references · 4
- 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
- 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
- 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
- 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Uglycited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
- 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
- 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
- 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
- 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
- 2026Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Models
- 2024Can We Talk Models Into Seeing the World Differently?
- 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding