mlsanityData to 14 Oct 2026

Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports

arXiv 2509.24298 ↗First posted 29 Sep 2025cs.HC cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The ability to represent emotion plays a significant role in human cognition and social interaction, yet the high-dimensional geometry of this affective space and its neural underpinnings remain debated. A key challenge, the `behavior-neural gap,' is the limited ability of human self-reports to predict brain activity. Here we test the hypothesis that this gap arises from the constraints of traditional rating scales and that large-scale similarity judgments can more faithfully capture the brain's affective geometry. Using AI models as `cognitive agents,' we collected millions of triplet odd-one-out judgments from a multimodal large language model (MLLM) and a language-only model (LLM) in response to 2,180 emotionally evocative videos. We found that the emergent 30-dimensional embeddings from these models are highly interpretable and organize emotion primarily along categorical lines, yet in a blended fashion that incorporates dimensional properties. Most remarkably, the MLLM's representation predicted neural activity in human emotion-processing networks with the highest accuracy, outperforming not only the LLM but also, counterintuitively, representations derived directly from human behavioral ratings. This result supports our primary hypothesis and suggests that sensory grounding--learning from rich visual data--is critical for developing a truly neurally-aligned conceptual framework for emotion. Our findings provide compelling evidence that MLLMs can autonomously develop rich, neurally-aligned affective representations, offering a powerful paradigm to bridge the gap between subjective experience and its neural substrates. Project page: https://reedonepeck.github.io/ai-emotion.github.io/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 112 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

21 of 112 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  9. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  10. 2023Scaling laws for language encoding models in fMRIcited by 68
  11. 2023Emotional Intelligence of Large Language Modelscited by 58
  12. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
9 more
  1. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 25
  2. 2024Dimensions underlying the representational alignment of deep neural networks with humanscited by 22
  3. 2019Revealing interpretable object representations from human behaviorcited by 17
  4. 2025AI with Emotions: Exploring Emotional Expressions in Large Language Modelscited by 11
  5. 2023The Good, The Bad, and Why: Unveiling Emotions in Generative AIcited by 10
  6. 2024Do Large Language Models Mirror Cognitive Language Processing?cited by 9
  7. 2023Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inferencecited by 6
  8. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  9. 2025Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Emotions Where Art Thou: Understanding and Characterizing the Emotional Latent Space of Large Language Models
  2. 2025Artificial Intelligence Can Emulate Human Normative Judgments on Emotional Visual Scenes ↗
  3. 2026Shared Emotion Geometry Across Small Language Models: A Cross-Architecture Study of Representation, Behavior, and Methodological Confounds
  4. 2026A Shared Valence Axis Across Modern LLMs and Human EEG: The Saturation Regularity
  5. 2025AI shares emotion with humans across languages and cultures
  6. 2024Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase
  7. 2026Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes
  8. 2024GPT-4 Emulates Average-Human Emotional Cognition from a Third-Person Perspective