mlsanityData to 14 Oct 2026

The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence

arXiv 2609.00868 ↗First posted 1 Sep 2026cs.CV cs.CL cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual input. We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks: blurring the question-relevant visual region leaves the next-token distribution nearly unchanged. We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI). The gap is a property of samples, not of models: VSI ranks correlate across models (grand-mean Spearman rho=+0.40, permutation p<10^-3), so the same samples are flagged insensitive by VLMs sharing no architectural detail beyond a contrastively pretrained vision tower. The mechanism is concrete: on the insensitive samples, a linear probe on each model's own vision tower distinguishes perturbed from clean images at 0.72--0.79 accuracy, yet the model's argmax token changes on only 2%--11% of the same samples, an encoder--LLM gap above 0.65 on every model. Mapping VSI's diagnostic utility cell by cell surfaces a strong regime (multi-choice reasoning on capable VLMs: AUROC=0.85--0.87) and a weak regime (well-calibrated factuality, where softmax confidence already leads). VSI is not a universal best abstention signal; it is a sample-intrinsic indicator of vision-ignoring failure, best used as a conditional ensemble component.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 31 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs20 Oct

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,353
10 more
  1. 2023Vision Transformers Need Registerscited by 871
  2. 2017Selective Classification for Deep Neural Networkscited by 852
  3. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  6. 2018Object Hallucination in Image Captioningcited by 669
  7. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  9. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  10. 2026Don't Blink: Evidence Collapse during Multimodal Reasoningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answeringcited by 1
  2. 2026When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Usedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  2. 2025Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations
  3. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  4. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  5. 2026Diagnosing Visual Ignorance in Vision-Language Models
  6. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
  7. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  8. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?