mlsanityData to 14 Oct 2026

EmoLLM: Multimodal Emotional Understanding Meets Large Language Models

arXiv 2406.16442 ↗First posted 24 Jun 2024cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Multi-modal large language models (MLLMs) have achieved remarkable performance on objective multimodal perception tasks, but their ability to interpret subjective, emotionally nuanced multimodal content remains largely unexplored. Thus, it impedes their ability to effectively understand and react to the intricate emotions expressed by humans through multimodal media. To bridge this gap, we introduce EmoBench, the first comprehensive benchmark designed specifically to evaluate the emotional capabilities of MLLMs across five popular emotional tasks, using a diverse dataset of 287k images and videos paired with corresponding textual instructions. Meanwhile, we propose EmoLLM, a novel model for multimodal emotional understanding, incorporating with two core techniques. 1) Multi-perspective Visual Projection, it captures diverse emotional cues from visual data from multiple perspectives. 2) EmoPrompt, it guides MLLMs to reason about emotions in the correct direction. Experimental results demonstrate that EmoLLM significantly elevates multimodal emotional understanding performance, with an average improvement of 12.1% across multiple foundation models on EmoBench. Our work contributes to the advancement of MLLMs by facilitating a deeper and more nuanced comprehension of intricate human emotions, paving the way for the development of artificial emotional intelligence capabilities with wide-ranging applications in areas such as human-computer interaction, mental health support, and empathetic AI systems. Code, data, and model will be released.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 55 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

23 of 55 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
11 more
  1. 2017Decoupled Weight Decay Regularizationcited by 1,831
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  6. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  7. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  8. 2023Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusioncited by 21
  9. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  10. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Modelcited by 12
  11. 2024Few-Shot VQA with Frozen LLMs: A Tale of Two Approachescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 7.

31 papers.

Sort:

By area:

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  2. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  3. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  4. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  6. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Surveycited by 19
  7. 2024Adaptive High-Frequency Transformer for Diverse Wildlife Re-Identificationcited by 10
  8. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 9
  9. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  10. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  11. 2024AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Modelscited by 7
  12. 2025FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoningcited by 5
19 more
  1. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  2. 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Modelscited by 5
  3. 2025EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibrationcited by 4
  4. 2025Commander-GPT: Fully Unleashing the Sarcasm Detection Capability of Multi-Modal Large Language Modelscited by 3
  5. 2025Towards Stable Cross-Domain Depression Recognition under Missing Modalitiescited by 3
  6. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  7. 2025Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detectioncited by 2
  8. 2025MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understandingcited by 2
  9. 2025Empathy-R1: A Chain-of-Empathy and Reinforcement Learning Framework for Long-Form Mental Health Supportcited by 2
  10. 2025Gems: Group Emotion Profiling Through Multimodal Situational Understandingcited by 1
  11. 2025Reasoning Like Experts: Leveraging Multimodal Large Language Models for Drawing-based Psychoanalysiscited by 1
  12. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Soundcited by 1
  13. 2026Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathycited by 1
  14. 2025Enhancing Collective Intelligence in Large Language Models Through Emotional Integrationcited by 0
  15. 2025Context-Emotion Aware Therapeutic Dialogue Generation: A Multi-component Reinforcement Learning Approach to Language Models for Mental Health Supportcited by 0
  16. 2026Towards Privacy-Preserving Mental Health Support with Large Language Modelscited by 0
  17. 2026ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoningcited by 0
  18. 2026ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoningcited by 0
  19. 2026ThinkDeception: A Progressive Reinforcement Learning Framework for Interpretable Multimodal Deception Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models
  2. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
  3. 2025Seeing is Not Understanding: A Benchmark on Perception-Cognition Disparities in Large Language Models
  4. 2025EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment
  5. 2026EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
  6. 2026EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness
  7. 2026From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding
  8. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding