mlsanityData to 14 Oct 2026

Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency

arXiv 2605.13047 ↗First posted 13 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,603 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias), and high-saliency objects. In contrast, models rely less on people in the scenes than our human participants do to describe the images. A model's size bias is a primary driver explaining variations in model-human semantic divergence. Code and data are available at https://github.com/starsky77/Counterfactual-Semantic-Saliency-Data .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 66 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

46 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations27 Sep

Other references · 45

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2025Qwen3-VL Technical Reportcited by 2,793
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
33 more
  1. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  2. 2025OpenAI GPT-5 System Cardcited by 989
  3. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  4. 2019Attention is not Explanationcited by 929
  5. 2025SAM 3: Segment Anything with Conceptscited by 906
  6. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  7. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  13. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  14. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  15. 2019Counterfactual Visual Explanationscited by 313
  16. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 218
  17. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 202
  18. 2021Partial success in closing the gap between human and machine visioncited by 185
  19. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  20. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 169
  21. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 165
  22. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  23. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  24. 2015Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labelscited by 64
  25. 2022Harmonizing the object recognition strategies of deep neural networks with humanscited by 59
  26. 2023ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classificationcited by 33
  27. 2024Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attentioncited by 17
  28. 2024Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Streamcited by 13
  29. 2024PooDLe: Pooled and dense self-supervised learning from naturalistic videoscited by 11
  30. 2025GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretabilitycited by 11
  31. 2025From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Modelscited by 7
  32. 2025V-CECE: Visual Counterfactual Explanations via Conceptual Editscited by 4
  33. 2023Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configurationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Semantic Saliency from Multi-Modal Large Language Model Scene Understanding Maps
  2. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
  3. 2025CF-VLM:CounterFactual Vision-Language Fine-tuning
  4. 2026SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
  5. 2025Vision-Language Models Can't See the Obvious
  6. 2022Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being?
  7. 2025CIVET: Systematic Evaluation of Understanding in VLMs
  8. 2025Predicting Reaction Time to Comprehend Scenes with Foveated Scene Understanding Maps