mlsanityData to 14 Oct 2026

Analyzing the Sensitivity of Vision Language Models in Visual Question Answering

arXiv 2507.21335 ↗First posted 28 Jul 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We can think of Visual Question Answering as a (multimodal) conversation between a human and an AI system. Here, we explore the sensitivity of Vision Language Models (VLMs) through the lens of cooperative principles of conversation proposed by Grice. Specifically, even when Grice's maxims of conversation are flouted, humans typically do not have much difficulty in understanding the conversation even though it requires more cognitive effort. Here, we study if VLMs are capable of handling violations to Grice's maxims in a manner that is similar to humans. Specifically, we add modifiers to human-crafted questions and analyze the response of VLMs to these modifiers. We use three state-of-the-art VLMs in our study, namely, GPT-4o, Claude-3.5-Sonnet and Gemini-1.5-Flash on questions from the VQA v2.0 dataset. Our initial results seem to indicate that the performance of VLMs consistently diminish with the addition of modifiers which indicates our approach as a promising direction to understand the limitations of VLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 21 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

6 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct

Other references · 5

  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  5. 2022End-to-End Transformer Based Model for Image Captioningcited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  2. 2026Context-Dependent Affordance Reports in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  2. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  3. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2024DARE: Diverse Visual Question Answering with Robustness Evaluation
  6. 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions
  7. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  8. 2026Seeing and Solving Are Not Enough for Vision-Language Models