mlsanityData to 7 Oct 2026

Contextual Emotion Recognition using Large Vision Language Models

arXiv 2405.08992 ↗First posted 14 May 2024cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

"How does the person in the bounding box feel?" Achieving human-level recognition of the apparent emotion of a person in real world situations remains an unsolved task in computer vision. Facial expressions are not enough: body pose, contextual knowledge, and commonsense reasoning all contribute to how humans perform this emotional theory of mind task. In this paper, we examine two major approaches enabled by recent large vision language models: 1) image captioning followed by a language-only LLM, and 2) vision language models, under zero-shot and fine-tuned setups. We evaluate the methods on the Emotions in Context (EMOTIC) dataset and demonstrate that a vision language model, fine-tuned even on a small dataset, can significantly outperform traditional baselines. The results of this work aim to help robots and agents perform emotionally sensitive decision-making and interaction in the future.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 67 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

39 of 67 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2020Longformer: The Long-Document Transformercited by 3,362
27 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2021MetaICL: Learning to Learn In Contextcited by 450
  11. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  12. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  13. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  14. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  15. 2019Context-Aware Emotion Recognition Networkscited by 99
  16. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 89
  17. 2020Context Based Emotion Recognition using EMOTIC Datasetcited by 73
  18. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
  19. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  20. 2020EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principlecited by 50
  21. 2023Context De-confounded Emotion Recognitioncited by 34
  22. 2021Global-Local Attention for Emotion Recognitioncited by 20
  23. 2022Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detectioncited by 14
  24. 2023High-Level Context Representation for Emotion Recognition in Imagescited by 7
  25. 2016From Dependence to Causationcited by 6
  26. 2022PERI: Part Aware Emotion Recognition In The Wildcited by 3
  27. 2023Contextual Emotion Estimation from Image Captionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognitioncited by 1
  2. 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?cited by 1
  3. 2026Investigating Associational Biases in Inter-Model Communication of Large Generative Modelscited by 1
  4. 2025MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledgecited by 0
  5. 2025E-THER: A Multimodal Dataset for Empathic AI -- Towards Emotional Mismatch Awarenesscited by 0
  6. 2026Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Modelscited by 0
  7. 2026A Controlled Study of CLIP-Based Body-Scene Fusion for Emotion Recognition in Contextcited by 0
  8. 2026Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning
  2. 2025Evaluating Vision-Language Models for Emotion Recognition
  3. 2026Why Do Vision Language Models Struggle To Recognize Human Emotions?
  4. 2026Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
  5. 2024Large Vision-Language Models as Emotion Recognizers in Context Awareness
  6. 2024VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning
  7. 2025Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models
  8. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach