mlsanityData to 14 Oct 2026

Vision-Language Models Can't See the Obvious

arXiv 2507.04741 ↗First posted 7 Jul 2025cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We present Saliency Benchmark (SalBench), a novel benchmark designed to assess the capability of Large Vision-Language Models (LVLM) in detecting visually salient features that are readily apparent to humans, such as a large circle amidst a grid of smaller ones. This benchmark focuses on low-level features including color, intensity, and orientation, which are fundamental to human visual processing. Our SalBench consists of images that highlight rare, unusual, or unexpected elements within scenes, and naturally draw human attention. It comprises three novel tasks for evaluating the perceptual capabilities of LVLM: Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out. We perform a comprehensive evaluation of state-of-the-art LVLM using SalBench and our findings reveal a surprising limitation: LVLM struggle to identify seemingly obvious visual anomalies, with even the advanced GPT-4o achieving only 47.6% accuracy on such a simple task. SalBench will be an important step in measuring the capabilities of LVLM that align with the subtle definition of human attention.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 57 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

43 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  2. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  3. 2024What matters when building vision-language models?3 May
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
26 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  15. 2016A Diagram Is Worth A Dozen Imagescited by 804
  16. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  17. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  18. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  19. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  20. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  21. 2018nocaps: novel object captioning at scalecited by 444
  22. 2019Scene Text Visual Question Answeringcited by 331
  23. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 160
  24. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  25. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  26. 2020Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluationscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Taskscited by 13
  2. 2025Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBenchcited by 7
  3. 2025MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessmentcited by 3
  4. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMscited by 3
  5. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 3
  6. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  7. 2026MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triagecited by 2
  8. 2026Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Rewardcited by 2
  9. 2026Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Taskcited by 1
  10. 2025Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learningcited by 0
  11. 2026Why MLLMs Struggle to Determine Object Orientationscited by 0
  12. 2026LADBench: A Benchmark for Logical Fault Detection in Imagescited by 0
2 more
  1. 2026ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Modelscited by 0
  2. 2026Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?
  2. 2024MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?
  3. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
  4. 2026Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
  5. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
  6. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models
  7. 2026Semantic Saliency from Multi-Modal Large Language Model Scene Understanding Maps
  8. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?