mlsanityData to 14 Oct 2026

Visual cognition in multimodal large language models

arXiv 2311.16093 ↗First posted 27 Nov 2023cs.LGcited by 38

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

A chief goal of artificial intelligence is to build machines that think like people. Yet it has been argued that deep neural network architectures fail to accomplish this. Researchers have asserted these models' limitations in the domains of causal reasoning, intuitive physics, and intuitive psychology. Yet recent advancements, namely the rise of large language models, particularly those designed for visual processing, have rekindled interest in the potential to emulate human-like cognitive abilities. This paper evaluates the current state of vision-based large language models in the domains of intuitive physics, causal reasoning, and intuitive psychology. Through a series of controlled experiments, we investigate the extent to which these modern models grasp complex physical interactions, causal relationships, and intuitive understanding of others' preferences. Our findings reveal that, while some of these models demonstrate a notable proficiency in processing and interpreting visual data, they still fall short of human capabilities in these areas. Our results emphasize the need for integrating more robust mechanisms for understanding causality, physical dynamics, and social cognition into modern-day, vision-based language models, and point out the importance of cognitively-inspired benchmarks.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 142 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

56 of 142 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models15 Nov
  2. 2023Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?31 Oct
  3. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
38 more
  1. 2016Generative Adversarial Text to Image Synthesiscited by 735
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  6. 2022Using cognitive psychology to understand GPT-3cited by 398
  7. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  8. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  9. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  10. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  11. 2023A Cookbook of Self-Supervised Learningcited by 250
  12. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  13. 2018Machine Theory of Mindcited by 226
  14. 2018Deep Learning: A Critical Appraisalcited by 217
  15. 2023Playing repeated games with Large Language Modelscited by 217
  16. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  17. 2021Partial success in closing the gap between human and machine visioncited by 185
  18. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  19. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 150
  20. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  21. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  22. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  23. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  24. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  25. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  26. 2023Turning large language models into cognitive modelscited by 67
  27. 2023Meta-in-context learning in large language modelscited by 59
  28. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
  29. 2019Causal Reasoning from Meta-reinforcement Learningcited by 53
  30. 2023AI-assisted coding: Experiments with GPT-4cited by 43
  31. 2019Theory of Minds: Understanding Behavior in Groups Through Inverse Planningcited by 30
  32. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Modelscited by 20
  33. 2022Towards Understanding How Machines Can Learn Causal Overhypothesescited by 14
  34. 2016Formalizing Neurath's Ship: Approximate Algorithms for Online Causal Learningcited by 12
  35. 2018Modeling human intuitions about liquid flow with particle-based simulationcited by 12
  36. 2023Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?cited by 10
  37. 2022Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mindcited by 9
  38. 2023Better by you, better than me, chatgpt3 as writing assistance in students essayscited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 14.

38 papers.

Sort:

By area:

  1. 2025Video models are zero-shot learners and reasonerscited by 220
  2. 2025Do generative video models understand physical principles?cited by 161
  3. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  4. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
  5. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  6. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  7. 2025Epistemological Fault Lines Between Human and Artificial Intelligencecited by 11
  8. 2025MimeQA: Towards Socially-Intelligent Nonverbal Foundation Modelscited by 9
  9. 2025The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Frameworkcited by 9
  10. 2025VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMscited by 7
  11. 2026Toward Cognitive Supersensing in Multimodal Large Language Modelcited by 7
  12. 2025Deep Learning in Concealed Dense Predictioncited by 6
26 more
  1. 2025Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Modelscited by 4
  2. 2025Decoupling the components of geometric understanding in Vision Language Modelscited by 4
  3. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  4. 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Modelscited by 4
  5. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 3
  6. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  7. 2025From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Imagescited by 2
  8. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modulescited by 2
  9. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 2
  10. 2025A Review on Large Language Models for Visual Analyticscited by 1
  11. 2025VisTopics: A Visual Semantic Unsupervised Approach to Topic Modeling of Video and Image Datacited by 1
  12. 2025CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updatescited by 1
  13. 2026AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflowscited by 1
  14. 2026Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memorycited by 1
  15. 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Modelcited by 0
  16. 2025Adaptive Token Boundaries: Integrating Human Chunking Mechanisms into Multimodal LLMscited by 0
  17. 2025Neuromorphic Computing with Multi-Frequency Oscillations: A Bio-Inspired Approach to Artificial Intelligencecited by 0
  18. 2025A biological vision inspired framework for machine perception of abutting grating illusory contourscited by 0
  19. 2025Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Modelcited by 0
  20. 2025Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detectioncited by 0
  21. 2026Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognitioncited by 0
  22. 2026LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")cited by 0
  23. 2026Mixture of Cognitive Experts in Large Vision-Language Modelscited by 0
  24. 2026A Diagnostic Framework for AI Agent Behaviorcited by 0
  25. 2026MonitorVLM-v2: A Deployed Vision-Language Framework for Real-Time Safety Violation Detectioncited by 0
  26. 2026CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
  2. 2025Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Models
  3. 2025Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis
  4. 2024Exploring Spatial Schema Intuitions in Large Language and Vision Models
  5. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
  6. 2026Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
  7. 2022Deep Neural Networks for Visual Reasoning
  8. 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models