mlsanityData to 14 Oct 2026

Core Knowledge Deficits in Multi-Modal Language Models

arXiv 2410.10855 ↗First posted 6 Oct 2024cs.CL cs.AI cs.CVcited by 38

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robustness in the wild remains limited, often falling short on tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core knowledge--rudimentary cognitive abilities innate to humans from early childhood. To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large-scale benchmark encompassing 12 core knowledge concepts grounded in developmental cognitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments uncover four key findings, collectively demonstrating core knowledge deficits in MLLMs: they consistently underperform and show reduced, or even absent, scalability on low-level abilities relative to high-level ones. Finally, we propose Concept Hacking, a novel controlled evaluation method that reveals MLLMs fail to progress toward genuine core knowledge understanding, but instead rely on shortcut learning as they scale.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 85 of its 240 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

85 of 240 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Data3 Oct
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024A Survey on Evaluation of Multimodal Large Language Models28 Aug
  4. 2024DevBench: A multimodal developmental benchmark for language learning14 Jun
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  6. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  7. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Models9 Feb
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  9. 2023Generative Multimodal Models are In-Context Learners20 Dec
  10. 2023VILA: On Pre-training for Visual Language Models12 Dec
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
1 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
38 more
  1. 2023Qwen Technical Reportcited by 3,344
  2. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  14. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  15. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  17. 2021Scaling Vision Transformerscited by 999
  18. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  19. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  20. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  21. 2023Instruction Tuning with GPT-4cited by 686
  22. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  23. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  24. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  25. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  26. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  27. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  28. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  29. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  30. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 280
  31. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  32. 2018Human Action Recognition and Prediction: A Surveycited by 227
  33. 2019Learning De-biased Representations with Biased Representationscited by 199
  34. 2023Teaching CLIP to Count to Tencited by 198
  35. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  36. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  37. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  38. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 22.

38 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  3. 2026A Very Big Video Reasoning Suitecited by 34
  4. 2026STEP3-VL-10B Technical Reportcited by 32
  5. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 29
  6. 2026Large Language Model Reasoning Failurescited by 21
  7. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 19
  8. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  9. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  10. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 13
  11. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  12. 2025Probing Perceptual Constancy in Large Vision-Language Modelscited by 7
26 more
  1. 2025The Philosophical Foundations of Growing AI Like A Childcited by 6
  2. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMscited by 5
  3. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 5
  4. 2025Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven' Matricescited by 4
  5. 2025BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Modelscited by 4
  6. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 4
  7. 2026Vision Language Models Cannot Reason About Physical Transformationcited by 4
  8. 2026Egocentric Bias in Vision-Language Modelscited by 3
  9. 2025Cognitive-Inspired Hierarchical Attention Fusion With Visual and Textual for Cross-Domain Sequential Recommendationcited by 2
  10. 2025Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoningcited by 2
  11. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modulescited by 2
  12. 2025Multimodal Evaluation of Russian-language Architecturescited by 1
  13. 2026BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encodercited by 1
  14. 2026Discovering Failure Modes in Vision-Language Models using RLcited by 1
  15. 2026SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenescited by 1
  16. 2026VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?cited by 1
  17. 2025SRNN: Spatiotemporal Relational Neural Network for Intuitive Physics Understandingcited by 0
  18. 2026KidVis: Do Multimodal Large Language Models Possess the Visual Perceptual Capabilities of a 6-Year-Old?cited by 0
  19. 2026History-Guided Iterative Visual Reasoning with Self-Correctioncited by 0
  20. 2026Minimal Embodiment Enables Efficient Learning of Number Concepts in Robotcited by 0
  21. 2026LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")cited by 0
  22. 2026ChronoVision: Temporal Reasoning via Latent State Reconstructioncited by 0
  23. 2026MindTopo: Can Foundation Models Reason in Topological Space?cited by 0
  24. 2026Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignmentcited by 0
  25. 2026Training Object Permanence in World Modelscited by 0
  26. 2026Gestalt: Large Multimodal Interplay Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Review of Developmental Interpretability in Large Language Models
  2. 2026BabyReasoningBench: Generating Developmentally-Inspired Reasoning Tasks for Evaluating Baby Language Models
  3. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models
  4. 2024The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks
  5. 2025Scaling Competence, Shrinking Reasoning: Cognitive Signatures in Language Model Learning
  6. 2025Uncovering the Computational Ingredients of Human-Like Representations in LLMs
  7. 2025Evidence of conceptual mastery in the application of rules by Large Language Models
  8. 2024FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition