mlsanityData to 14 Oct 2026

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models

arXiv 2501.16566 ↗First posted 27 Jan 2025cs.HCcited by 44

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

The emergence of multimodal large language models (MLLMs) advances multimodal emotion recognition (MER) to the next level, from naive discriminative tasks to complex emotion understanding with advanced video understanding abilities and natural language description. However, the current community suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations, as well as a multimodal-centric framework to maximize the potential of MLLMs for emotion understanding. To address this, we establish a new benchmark for MLLM-based emotion understanding with a novel dataset (MER-Caption) and a new model (AffectGPT). Utilizing our model-based crowd-sourcing data collection strategy, we construct the largest descriptive emotion dataset to date (by far), featuring over 2K fine-grained emotion categories across 115K samples. We also introduce the AffectGPT model, designed with pre-fusion operations to enhance multimodal integration. Finally, we present MER-UniBench, a unified benchmark with evaluation metrics tailored for typical MER tasks and the free-form, natural language output style of MLLMs. Extensive experimental results show AffectGPT's robust performance across various MER tasks. We have released both the code and the dataset to advance research and development in emotion understanding: https://github.com/zeroQiaoba/AffectGPT.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 58 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

37 of 58 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  12. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
25 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  3. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  4. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 492
  5. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  9. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  10. 2021Towards Emotional Support Dialog Systemscited by 299
  11. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  12. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  13. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  14. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  15. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  16. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
  17. 2022MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wildcited by 65
  18. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 44
  19. 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 43
  20. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  21. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  22. 2023SECap: Speech Emotion Captioning with Large Language Modelcited by 33
  23. 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
  24. 2018Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignmentcited by 20
  25. 2024ECR-Chain: Advancing Generative Language Models to Better Emotion-Cause Reasoners through Reasoning Chainscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 31.

44 papers.

Sort:

By area:

  1. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  2. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  3. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  4. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  5. 2025EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Modelscited by 7
  6. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  7. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  8. 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Modelscited by 5
  9. 2025EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Languagecited by 4
  10. 2025EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibrationcited by 4
  11. 2026EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessmentcited by 4
  12. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 4
32 more
  1. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  2. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approachcited by 3
  3. 2025Towards Stable Cross-Domain Depression Recognition under Missing Modalitiescited by 3
  4. 2026Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventionscited by 3
  5. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Soundcited by 1
  6. 2026Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervalscited by 1
  7. 2026XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoningcited by 1
  8. 2026Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathycited by 1
  9. 2026EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMscited by 1
  10. 2026MOTOR-Bench: A Real-world Dataset and Multi-agent Framework for Zero-shot Human Mental State Understandingcited by 1
  11. 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoningcited by 1
  12. 2026Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approachcited by 1
  13. 2026Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaborationcited by 1
  14. 2026Benchmarking Dynamic Affective Reasoning: A Viewer-Centric Video Emotion Datasetcited by 1
  15. 2025Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captionscited by 0
  16. 2025Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundariescited by 0
  17. 2026Understanding Affective Adaptation in Multimodal Foundation Models: Emergent Functional Specializationcited by 0
  18. 2026ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoningcited by 0
  19. 2026AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognitioncited by 0
  20. 2026AffectSeek: Agentic Affective Understanding in Long Videos under Vague User Queriescited by 0
  21. 2026Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Modelscited by 0
  22. 2026Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flowcited by 0
  23. 2026Faithful Action-unit Causal Reasoning for Counterfactually Faithful Emotion Explanationscited by 0
  24. 2026COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intentioncited by 0
  25. 2026RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomycited by 0
  26. 2026Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observationscited by 0
  27. 2026Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Groundingcited by 0
  28. 2026AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearancescited by 0
  29. 2026DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoningcited by 0
  30. 2026Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruningcited by 0
  31. 2026LongEmo: Towards Emotion Understanding and Reasoning in Long Videoscited by 0
  32. 2026EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies
  2. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  3. 2026Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?
  4. 2025MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs
  5. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
  6. 2025Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
  7. 2026MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
  8. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models