mlsanityData to 14 Oct 2026

Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities

arXiv 2507.10442 ↗First posted 10 Jul 2025cs.LG cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems. Such models have been shown to be highly capable, but, at the same time, lacking some basic visual understanding skills. In this paper, we set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking. Importantly, we go significantly beyond the current benchmarks, which simply measure the final performance of VLM response, by also comparing and contrasting it to the performance of probes trained directly on features obtained from the visual encoder, intermediate vision-language projection and LLM-decoder output. In doing so, we uncover shortcomings in VLMs and make a number of important observations about their capabilities, robustness and how they process visual information. We hope our insights will guide progress in further improving VLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 51 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

27 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)7 Oct
  2. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions29 May
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding30 Nov
  4. 2023Teaching CLIP to Count to Ten23 Feb

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2019Designing and Interpreting Probes with Control Taskscited by 640
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  11. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  12. 2017Scene Graph Generation by Iterative Message Passingcited by 462
11 more
  1. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 336
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  3. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  4. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  5. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  6. 2016Richer Convolutional Features for Edge Detectioncited by 143
  7. 2018Benchmarking Neural Network Robustness to Common Corruptions and Surface Variationscited by 99
  8. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 89
  9. 2025Vision-Language Models Do Not Understand Negationcited by 72
  10. 2021Does enhanced shape bias improve neural network robustness to common corruptions?cited by 32
  11. 2024ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities
  2. 2025Hidden in plain sight: VLMs overlook their visual representations
  3. 2024An Introduction to Vision-Language Modeling
  4. 2024What matters when building vision-language models?
  5. 2023Vision-Language Models for Vision Tasks: A Survey
  6. 2025A Survey on Efficient Vision-Language Models
  7. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  8. 2025Visual Large Language Models for Generalized and Specialized Applications