mlsanityData to 14 Oct 2026

What is the Visual Cognition Gap between Humans and Multimodal LLMs?

arXiv 2406.10424 ↗First posted 14 Jun 2024cs.CV cs.AIcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, Multimodal Large Language Models (MLLMs) and Vision Language Models (VLMs) have shown great promise in language-guided perceptual tasks such as recognition, segmentation, and object detection. However, their effectiveness in addressing visual cognition problems that require high-level multi-image reasoning and visual working memory is not well-established. One such challenge is matrix reasoning - the cognitive ability to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This skill is crucial during the early neurodevelopmental stages of children. Inspired by the matrix reasoning tasks in Raven's Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC), we propose a new dataset MaRs-VQA to evaluate the visual cognition capability of MLLMs and compare their performance with existing human visual cognition studies. Based on the training data of MaRs-VQA, we also finetune a baseline model Qwen2-VCog with multi-stage cognition reasoning annotations. Our comparative experiments with different baselines reveal a gap between MLLMs and human intelligence, highlighting the visual cognitive limitations of current MLLMs. We believe that the public release of MaRs-VQA and the Qwen2-VCog baseline model will drive progress toward the next generation of MLLMs with human-like visual cognition abilities. MaRs-VQA is available at huggingface.co/datasets/IrohXu/VCog-Bench. The training code of Qwen2-VCog is available at github.com/IrohXu/Cognition-MLLM.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 107 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

63 of 107 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  4. 2024The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models22 Jan
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Mistral 7Bcited by 3,164
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
38 more
  1. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  4. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  6. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  10. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  11. 2022Using cognitive psychology to understand GPT-3cited by 398
  12. 2024InternLM2 Technical Reportcited by 397
  13. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  14. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  15. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  16. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  17. 2024An Introduction to Vision-Language Modelingcited by 178
  18. 2018Measuring abstract reasoning in neural networkscited by 166
  19. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  20. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 101
  21. 2023Object-Centric Slot Diffusioncited by 90
  22. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 82
  23. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  24. 2020Stratified Rule-Aware Network for Abstract Visual Reasoningcited by 72
  25. 2021Learning to Compose Visual Relationscited by 72
  26. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  27. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 65
  28. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  29. 2020Emergent Symbols through Binding in External Memorycited by 47
  30. 2020Scale-Localized Abstract Reasoningcited by 45
  31. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  32. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  33. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  34. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  35. 2024How Far Are We from Intelligent Visual Deductive Reasoning?cited by 29
  36. 2023GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language modelscited by 28
  37. 2023The Relational Bottleneck as an Inductive Bias for Efficient Abstractioncited by 27
  38. 2023Systematic Visual Reasoning through Object-Centric Relational Abstractioncited by 20

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 11.

30 papers.

Sort:

By area:

  1. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2024Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problemscited by 15
  4. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  5. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understandingcited by 12
  6. 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 12
  7. 2024Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoningcited by 7
  8. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  9. 2025Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problemscited by 7
  10. 2026Toward Cognitive Supersensing in Multimodal Large Language Modelcited by 7
  11. 2024The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarkscited by 6
  12. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddingscited by 6
18 more
  1. 2025A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMscited by 5
  2. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5
  3. 2024Measuring Agreeableness Bias in Multimodal Modelscited by 4
  4. 2025Caption This, Reason That: VLMs Caught in the Middlecited by 4
  5. 2025I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Modelscited by 4
  6. 2026Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defensecited by 3
  7. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  8. 2025Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluationcited by 2
  9. 2025SPHINX: A Synthetic Environment for Visual Perception and Reasoningcited by 2
  10. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzlescited by 2
  11. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  12. 2026Evaluating Cognitive Age Alignment in Interactive AI Agentscited by 2
  13. 2026A Dialogue-Based Framework for Correcting Multimodal Errors in AI-Assisted STEM Educationcited by 1
  14. 2026Thinking in Pictures: A Systematic Benchmark for Reasoning-driven Image Generationcited by 1
  15. 2025Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoningcited by 0
  16. 2025The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?cited by 0
  17. 2026LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")cited by 0
  18. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BabyVision: Visual Reasoning Beyond Language
  2. 2026KidVis: Do Multimodal Large Language Models Possess the Visual Perceptual Capabilities of a 6-Year-Old?
  3. 2025Visual Language Models show widespread visual deficits on neuropsychological tests
  4. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
  5. 2025Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis
  6. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  7. 2025MME-CC: A Challenging Multi-Modal Evaluation Benchmark of Cognitive Capacity
  8. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?