mlsanityData to 7 Oct 2026

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

arXiv 2406.11161 ↗First posted 17 Jun 2024cs.AI cs.MMcited by 117

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 69 of its 104 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

69 of 104 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  13. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  14. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  15. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  22. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  23. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  24. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  25. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  26. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  27. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  28. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  29. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  30. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  31. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  32. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  33. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  34. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  35. 2020Suppressing Uncertainties for Large-Scale Facial Expression Recognitioncited by 157
  36. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  37. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 81
  38. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 50; 2026: 57.

117 papers.

Sort:

By area:

  1. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  2. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  3. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  4. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  5. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  6. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  7. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  8. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  9. 2024Large Vision-Language Models as Emotion Recognizers in Context Awarenesscited by 12
  10. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  11. 2025Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmarkcited by 12
  12. 2025FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMscited by 11
38 more
  1. 2026PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebracited by 11
  2. 2024Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wildcited by 9
  3. 2025Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuningcited by 9
  4. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  5. 2024StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Modelscited by 8
  6. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  7. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  8. 2025OmniVox: Zero-Shot Emotion Recognition with Omni-LLMscited by 7
  9. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  10. 2025FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPOcited by 6
  11. 2026AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognitioncited by 6
  12. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  13. 2025VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injectioncited by 5
  14. 2025FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoningcited by 5
  15. 2025MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detectioncited by 5
  16. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  17. 2025EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Languagecited by 4
  18. 2025More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignmentcited by 4
  19. 2025TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognitioncited by 4
  20. 2026CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learningcited by 4
  21. 2024EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understandingcited by 3
  22. 2025Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approachescited by 3
  23. 2025Linguistic Knowledge Transfer Learning for Speech Enhancementcited by 3
  24. 2025MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labelingcited by 3
  25. 2025Towards Reliable Large Audio Language Modelcited by 3
  26. 2025MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Modelscited by 3
  27. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Promptingcited by 3
  28. 2025Multimodal Video Emotion Recognition with Reliable Reasoning Priorscited by 3
  29. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  30. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approachcited by 3
  31. 2025Towards Stable Cross-Domain Depression Recognition under Missing Modalitiescited by 3
  32. 2025EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibrationcited by 3
  33. 2026VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentationcited by 3
  34. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 3
  35. 2024POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Searchcited by 2
  36. 2025LLMPopcorn: Exploring LLMs as Assistants for Popular Micro-video Generationcited by 2
  37. 2025MELLM: A Flow-Guided Large Language Model for Micro-Expression Understandingcited by 2
  38. 2025Are MLMs Trapped in the Visual Room?cited by 2

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  2. 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies
  3. 2025Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier
  4. 2025Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis
  5. 2025DEEMO: De-identity Multimodal Emotion Recognition and Reasoning
  6. 2025Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding
  7. 2026Navigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognition
  8. 2026MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition