Evaluating VQA in Vision Language Models using Cooperative Principles
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs to generate question modifiers that add non-essential, ambiguous or false information and show that in the presence of such violations, the VLMs that we evaluate (ChatGPT, Claude, Gemini and Llava) show diminished performance. Further, we empirically show the difference between how humans reason pragmatically compared to VLMs, and the difference in VLM reasoning when it resolves violations that are human-induced compared to those that are AI-generated. Finally, we show that human cognitive effort (measured through time-on-task in an experiment) is lower for resolving VLM-induced violations, but VLMs themselves perform less accurately in such cases.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 2 Oct 2026.
Builds on
No reference list yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Analyzing the Sensitivity of Vision Language Models in Visual Question Answering
- 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
- 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
- 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?
- 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions
- 2025How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmark
- 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
- 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering