mlsanityData to 14 Oct 2026

Visual Language Models show widespread visual deficits on neuropsychological tests

arXiv 2504.10786 ↗First posted 15 Apr 2025cs.CV cs.AI cs.LGcited by 3

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Visual Language Models (VLMs) show remarkable performance in visual reasoning tasks, successfully tackling college-level challenges that require high-level understanding of images. However, some recent reports of VLMs struggling to reason about elemental visual concepts like orientation, position, continuity, and occlusion suggest a potential gulf between human and VLM vision. Here we use the toolkit of neuropsychology to systematically assess the capabilities of three state-of-the-art VLMs across visual domains. Using 51 tests drawn from six clinical and experimental batteries, we characterise the visual abilities of leading VLMs relative to normative performance in healthy adults. While the models excel in straightforward object recognition tasks, we find widespread deficits in low- and mid-level visual abilities that would be considered clinically significant in humans. These selective deficits, profiled through validated test batteries, suggest that an artificial system can achieve complex object recognition without developing foundational visual concepts that in humans require no explicit training.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 83 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

44 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding17 Feb
  2. 2024VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Models23 Sep
  3. 2024Visually Descriptive Language Model for Vector Graphics Reasoning9 Apr
  4. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
27 more
  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  4. 2022Using cognitive psychology to understand GPT-3cited by 398
  5. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  6. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  7. 2024An Introduction to Vision-Language Modelingcited by 178
  8. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  9. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  10. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
  11. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  12. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  13. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
  14. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 47
  15. 2023Neither hype nor gloom do DNNs justicecited by 40
  16. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  17. 2023Visual cognition in multimodal large language modelscited by 38
  18. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 35
  19. 2023Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?cited by 32
  20. 2024Exploring Perceptual Limitation of Multimodal Large Language Modelscited by 25
  21. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Informationcited by 16
  22. 2024Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiadscited by 16
  23. 2024MindSet: Vision. A toolbox for testing DNNs on key psychological experimentscited by 8
  24. 2024Understanding Depth and Height Perception in Large Visual-Language Modelscited by 8
  25. 2024Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Studycited by 5
  26. 2024OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objectscited by 3
  27. 2024OCC-MLLM-Alpha:Empowering Multi-modal Large Language Model for the Understanding of Occluded Objects with Self-Supervised Test-Time Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understandingcited by 2
  2. 2026Using profiles of cognitive capability to assess AI suitability for workplace taskscited by 0
  3. 2026Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Caption This, Reason That: VLMs Caught in the Middle
  2. 2025Visual serial processing deficits explain divergences in human and VLM reasoning
  3. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
  4. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  5. 2025VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
  6. 2025Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models
  7. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?
  8. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs