mlsanityData to 7 Oct 2026

Human-like object concept representations emerge naturally in multimodal large language models

arXiv 2407.01067 ↗First posted 1 Jul 2024cs.AI cs.CL cs.CVcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Understanding how humans conceptualize and categorize natural objects offers critical insights into perception and cognition. With the advent of Large Language Models (LLMs), a key question arises: can these models develop human-like object representations from linguistic and multimodal data? In this study, we combined behavioral and neuroimaging analyses to explore the relationship between object concept representations in LLMs and human cognition. We collected 4.7 million triplet judgments from LLMs and Multimodal LLMs (MLLMs) to derive low-dimensional embeddings that capture the similarity structure of 1,854 natural objects. The resulting 66-dimensional embeddings were stable, predictive, and exhibited semantic clustering similar to human mental representations. Remarkably, the dimensions underlying these embeddings were interpretable, suggesting that LLMs and MLLMs develop human-like conceptual representations of objects. Further analysis showed strong alignment between model embeddings and neural activity patterns in brain regions such as EBA, PPA, RSC, and FFA. This provides compelling evidence that the object representations in LLMs, while not identical to human ones, share fundamental similarities that reflect key aspects of human conceptual knowledge. Our findings advance the understanding of machine intelligence and inform the development of more human-like artificial cognitive systems.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 124 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

37 of 124 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2024Large Language Models: A Surveycited by 506
  11. 2022Using cognitive psychology to understand GPT-3cited by 389
  12. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
24 more
  1. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  2. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  3. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  4. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  5. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  6. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  7. 2022Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolutioncited by 78
  8. 2022Human alignment of neural network representationscited by 74
  9. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  10. 2023Large language models predict human sensory judgments across six modalitiescited by 47
  11. 2023Neither hype nor gloom do DNNs justicecited by 40
  12. 2024Dimensions underlying the representational alignment of deep neural networks with humanscited by 22
  13. 2019Revealing interpretable object representations from human behaviorcited by 17
  14. 2020Extracting low-dimensional psychological representations from convolutional neural networkscited by 16
  15. 2023Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language modelscited by 16
  16. 2019Learning as the Unsupervised Alignment of Conceptual Systemscited by 14
  17. 2023Aspects of human memory and Large Language Modelscited by 12
  18. 2023Information decomposition in complex systems via machine learningcited by 9
  19. 2024Integration of cognitive tasks into artificial general intelligence test for large modelscited by 5
  20. 2024CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representationscited by 5
  21. 2023Divergences in Color Perception between Deep Neural Networks and Humanscited by 4
  22. 2024Human-Like Geometric Abstraction in Large Pre-trained Neural Networkscited by 4
  23. 2024Towards a Psychology of Machines: Large Language Models Predict Human Memorycited by 4
  24. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 13.

24 papers.

Sort:

By area:

  1. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
  2. 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 7
  3. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  4. 2025Language models align with brain regions that represent concepts across modalitiescited by 4
  5. 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 3
  6. 2024Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligencecited by 2
  7. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  8. 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysiscited by 1
  9. 2025Inducing Dyslexia in Vision Language Modelscited by 1
  10. 2026SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstructioncited by 1
  11. 2026Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoningcited by 1
  12. 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Modelcited by 0
12 more
  1. 2025Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reportscited by 0
  2. 2025Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddingscited by 0
  3. 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystemcited by 0
  4. 2025A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networkscited by 0
  5. 2025Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wildcited by 0
  6. 2026Large language models for spreading dynamics in complex systemscited by 0
  7. 2026Similarity-based representation factorization for revealing interpretable dimensions in representational datacited by 0
  8. 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0
  9. 2026Reward Valuation in Large Language Models: Causal Induction of Anhedoniacited by 0
  10. 2026Lexical discovery in unknown environments orchestrated by Large Language Modelscited by 0
  11. 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognitioncited by 0
  12. 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Uncovering the Computational Ingredients of Human-Like Representations in LLMs
  2. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  3. 2025Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
  4. 2022Visual representations in the human brain are aligned with large language models
  5. 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
  6. 2024Large Multi-modal Models Can Interpret Features in Large Multi-modal Models
  7. 2026Would you still call this Dax? Novel Visual References in VLMs and Humans
  8. 2024Identifying and interpreting non-aligned human conceptual representations using language modeling