Human-like object concept representations emerge naturally in multimodal large language models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Understanding how humans conceptualize and categorize natural objects offers critical insights into perception and cognition. With the advent of Large Language Models (LLMs), a key question arises: can these models develop human-like object representations from linguistic and multimodal data? In this study, we combined behavioral and neuroimaging analyses to explore the relationship between object concept representations in LLMs and human cognition. We collected 4.7 million triplet judgments from LLMs and Multimodal LLMs (MLLMs) to derive low-dimensional embeddings that capture the similarity structure of 1,854 natural objects. The resulting 66-dimensional embeddings were stable, predictive, and exhibited semantic clustering similar to human mental representations. Remarkably, the dimensions underlying these embeddings were interpretable, suggesting that LLMs and MLLMs develop human-like conceptual representations of objects. Further analysis showed strong alignment between model embeddings and neural activity patterns in brain regions such as EBA, PPA, RSC, and FFA. This provides compelling evidence that the object representations in LLMs, while not identical to human ones, share fundamental similarities that reflect key aspects of human conceptual knowledge. Our findings advance the understanding of machine intelligence and inform the development of more human-like artificial cognitive systems.
Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 124 references that are in the map. First posted on arXiv 1 Jul 2024.
Builds on
37 of 124 references.
Closest ancestors · 1
Same direction, newest first
Other references · 36
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2023A Survey on Evaluation of Large Language Modelscited by 1,227
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2024Large Language Models: A Surveycited by 506
- 2022Using cognitive psychology to understand GPT-3cited by 389
- 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
24 more
- 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
- 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
- 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
- 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
- 2022Thinking Fast and Slow in Large Language Modelscited by 136
- 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
- 2022Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolutioncited by 78
- 2022Human alignment of neural network representationscited by 74
- 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
- 2023Large language models predict human sensory judgments across six modalitiescited by 47
- 2023Neither hype nor gloom do DNNs justicecited by 40
- 2024Dimensions underlying the representational alignment of deep neural networks with humanscited by 22
- 2019Revealing interpretable object representations from human behaviorcited by 17
- 2020Extracting low-dimensional psychological representations from convolutional neural networkscited by 16
- 2023Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language modelscited by 16
- 2019Learning as the Unsupervised Alignment of Conceptual Systemscited by 14
- 2023Aspects of human memory and Large Language Modelscited by 12
- 2023Information decomposition in complex systems via machine learningcited by 9
- 2024Integration of cognitive tasks into artificial general intelligence test for large modelscited by 5
- 2024CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representationscited by 5
- 2023Divergences in Color Perception between Deep Neural Networks and Humanscited by 4
- 2024Human-Like Geometric Abstraction in Large Pre-trained Neural Networkscited by 4
- 2024Towards a Psychology of Machines: Large Language Models Predict Human Memorycited by 4
- 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questionscited by 4
Built on by
Citing papers per year: 2024: 2; 2025: 9; 2026: 13.
24 papers.
Sort:
By area:
- 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
- 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 7
- 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
- 2025Language models align with brain regions that represent concepts across modalitiescited by 4
- 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 3
- 2024Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligencecited by 2
- 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
- 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysiscited by 1
- 2025Inducing Dyslexia in Vision Language Modelscited by 1
- 2026SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstructioncited by 1
- 2026Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoningcited by 1
- 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Modelcited by 0
12 more
- 2025Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reportscited by 0
- 2025Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddingscited by 0
- 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystemcited by 0
- 2025A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networkscited by 0
- 2025Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wildcited by 0
- 2026Large language models for spreading dynamics in complex systemscited by 0
- 2026Similarity-based representation factorization for revealing interpretable dimensions in representational datacited by 0
- 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0
- 2026Reward Valuation in Large Language Models: Causal Induction of Anhedoniacited by 0
- 2026Lexical discovery in unknown environments orchestrated by Large Language Modelscited by 0
- 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognitioncited by 0
- 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language modelscited by 0
- 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models10 Sep 2026
- 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition7 Sep 2026
- 2026Reward Valuation in Large Language Models: Causal Induction of Anhedonia7 Jul 2026
- 2026Lexical discovery in unknown environments orchestrated by Large Language Models10 Jun 2026
- 2026Beyond representational alignment with brain-guided language models for robust reasoning10 Jun 2026
- 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMs1 Jun 2026
- 2026Similarity-based representation factorization for revealing interpretable dimensions in representational data26 May 2026
- 2026Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning15 Apr 2026
- 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding19 Mar 2026
- 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs2 Mar 2026
- 2026Causality neq Invariance: Function and Concept Vectors in LLMs25 Feb 2026
- 2026Large language models for spreading dynamics in complex systems8 Feb 2026
12 more
- 2026SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction25 Jan 2026
- 2025Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild4 Dec 2025
- 2025A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks28 Oct 2025
- 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystem24 Oct 2025
- 2025Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddings10 Oct 2025
- 2025Inducing Dyslexia in Vision Language Models29 Sep 2025
- 2025Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports29 Sep 2025
- 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents9 Sep 2025
- 2025Language models align with brain regions that represent concepts across modalities15 Aug 2025
- 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Model5 Mar 2025
- 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysis30 Nov 2024
- 2024Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligence4 Jul 2024
- 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
- 2025Inducing Dyslexia in Vision Language Modelscited by 1
- 2025Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wildcited by 0
- 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0
- 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language modelscited by 0
- 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models10 Sep 2026
- 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMs1 Jun 2026
- 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding19 Mar 2026
- 2025Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild4 Dec 2025
- 2025Inducing Dyslexia in Vision Language Models29 Sep 2025
- 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 3
- 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
- 2026Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoningcited by 1
- 2026Beyond representational alignment with brain-guided language models for robust reasoning10 Jun 2026
- 2026Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning15 Apr 2026
- 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs2 Mar 2026
- 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 7
- 2025Language models align with brain regions that represent concepts across modalitiescited by 4
- 2024Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligencecited by 2
- 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysiscited by 1
- 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Modelcited by 0
- 2025Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reportscited by 0
- 2025Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddingscited by 0
- 2025A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networkscited by 0
- 2026Large language models for spreading dynamics in complex systemscited by 0
- 2026Similarity-based representation factorization for revealing interpretable dimensions in representational datacited by 0
- 2026Reward Valuation in Large Language Models: Causal Induction of Anhedoniacited by 0
- 2026Lexical discovery in unknown environments orchestrated by Large Language Modelscited by 0
12 most cited of 13 All 13 (JSON)
- 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition7 Sep 2026
- 2026Reward Valuation in Large Language Models: Causal Induction of Anhedonia7 Jul 2026
- 2026Lexical discovery in unknown environments orchestrated by Large Language Models10 Jun 2026
- 2026Similarity-based representation factorization for revealing interpretable dimensions in representational data26 May 2026
- 2026Causality neq Invariance: Function and Concept Vectors in LLMs25 Feb 2026
- 2026Large language models for spreading dynamics in complex systems8 Feb 2026
- 2025A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks28 Oct 2025
- 2025Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddings10 Oct 2025
- 2025Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports29 Sep 2025
- 2025Language models align with brain regions that represent concepts across modalities15 Aug 2025
- 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Model5 Mar 2025
- 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysis30 Nov 2024
12 newest of 13 All 13 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2025Uncovering the Computational Ingredients of Human-Like Representations in LLMs
- 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
- 2025Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
- 2022Visual representations in the human brain are aligned with large language models
- 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
- 2024Large Multi-modal Models Can Interpret Features in Large Multi-modal Models
- 2026Would you still call this Dax? Novel Visual References in VLMs and Humans
- 2024Identifying and interpreting non-aligned human conceptual representations using language modeling