mlsanityData to 7 Oct 2026

EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models

arXiv 2502.04424 ↗First posted 6 Feb 2025cs.CL cs.AIcited by 32

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

With the integration of multimodal large language models (MLLMs) into robotic systems and AI applications, embedding emotional intelligence (EI) capabilities is essential for enabling these models to perceive, interpret, and respond to human emotions effectively in real-world scenarios. Existing static, text-based, or text-image benchmarks overlook the multimodal complexities of real interactions and fail to capture the dynamic, context-dependent nature of emotional expressions, rendering them inadequate for evaluating MLLMs' EI capabilities. To address these limitations, we introduce EmoBench-M, a systematic benchmark grounded in established psychological theories, designed to evaluate MLLMs across 13 evaluation scenarios spanning three hierarchical dimensions: foundational emotion recognition (FER), conversational emotion understanding (CEU), and socially complex emotion analysis (SCEA). Evaluation was conducted on 27 state-of-the-art MLLMs, using both objective task-specific metrics and LLM-based evaluation, revealing a substantial performance gap relative to human-level competence. Even the best performing models, Gemini-3.0-Pro and GPT-5.2, achieve the highest scores on EmoBench-M, 70.5 and 66.5 points respectively. Specialized models such as AffectGPT exhibit uneven performance across EmoBench-M, demonstrating strengths in certain scenarios but generally lacking comprehensive emotional intelligence. By providing a comprehensive, multimodal evaluation framework, EmoBench-M captures both the strengths and weaknesses of current MLLMs across diverse emotional contexts. All benchmark resources, including datasets and code, are publicly available at https://emo-gml.github.io/, facilitating further research and advancement in MLLM emotional intelligence.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 74 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

45 of 74 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models27 Jan

Other references · 44

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  12. 2024Qwen2-Audio Technical Reportcited by 543
32 more
  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2024Long Context Transfer from Language to Visioncited by 502
  3. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  4. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  5. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  6. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  7. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  8. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  9. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  10. 2019Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advancescited by 118
  11. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  12. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  13. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  14. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  15. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  16. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
  17. 2023Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvementscited by 42
  18. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
  19. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  20. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
  21. 2024MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognitioncited by 27
  22. 2024M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Modelscited by 26
  23. 2024LearnLM: Improving Gemini for Learningcited by 24
  24. 2024SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understandingcited by 20
  25. 2024OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognitioncited by 17
  26. 2024Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Datasetcited by 15
  27. 2023Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysiscited by 14
  28. 2024Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligencecited by 11
  29. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Taskscited by 11
  30. 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
  31. 2023SOUL: Towards Sentiment and Opinion Understanding of Languagecited by 4
  32. 2024Impromptu Cybercrime Euphemism Detectioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 19.

32 papers.

Sort:

By area:

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Surveycited by 19
  3. 2025Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counselingcited by 17
  4. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  5. 2025VisNumBench: Evaluating Number Sense of Multimodal Large Language Modelscited by 12
  6. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  7. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  8. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  9. 2026The ICASSP 2026 HumDial Challenge: Benchmarking Human-like Spoken Dialogue Systems in the LLM Eracited by 5
  10. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  11. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Modelscited by 4
  12. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approachcited by 3
20 more
  1. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guaranteescited by 3
  2. 2025Contrastive Speaker-Aware Learning for Multi-party Dialogue Generation with LLMscited by 2
  3. 2025High-Fidelity Pseudo-label Generation by Large Language Models for Training Robust Radiology Report Classifierscited by 2
  4. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  5. 2026Mental-R1: Aligning LLM Reasoning for Mental Health Assessmentcited by 2
  6. 2025Generating Medically-Informed Explanations for Depression Detection using LLMscited by 1
  7. 2026XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoningcited by 1
  8. 2026EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMscited by 1
  9. 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoningcited by 1
  10. 2026ESC: Emotional Self-Correction for Reliable Vision-Language Modelscited by 1
  11. 2026MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgementcited by 1
  12. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMscited by 0
  13. 2025Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translationcited by 0
  14. 2025Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performancecited by 0
  15. 2026AE-LLM: Adaptive Efficiency Optimization for Large Language Modelscited by 0
  16. 2026MIThinker: A Plug-and-Play Policy-Optimized Thinker For Motivational Interviewing Counselingcited by 0
  17. 2026The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMscited by 0
  18. 2026From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Healthcited by 0
  19. 2026LongEmo: Towards Emotion Understanding and Reasoning in Long Videoscited by 0
  20. 2026Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Models
  2. 2025Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
  3. 2026EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management
  4. 2025LongEmotion: Measuring Emotional Intelligence of Large Language Models in Long-Context Interaction
  5. 2026EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
  6. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning
  7. 2025EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment
  8. 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies