# Evaluating Vision-Language Models for Emotion Recognition

arXiv: https://arxiv.org/abs/2502.05660 · HTML: https://mlsanity.com/paper/2502.05660/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Multimodal LLMs / vision-language models / [Multimodal emotion understanding](https://mlsanity.com/mllm/multimodal-emotion-understanding/)

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems](https://mlsanity.com/paper/2204.03021/) (2022)
- [EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes](https://mlsanity.com/paper/2307.07961/) (2023)
- [Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark](https://mlsanity.com/paper/1605.02677/) (2016)
- [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://mlsanity.com/paper/2402.09369/) (2024)
- [EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis](https://mlsanity.com/paper/2401.08508/) (2024)
- [EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principle](https://mlsanity.com/paper/2003.06692/) (2020)
- [EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning](https://mlsanity.com/paper/2404.16670/) (2024)
- [Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought](https://mlsanity.com/paper/2401.06836/) (2024)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Stimuli-Aware Visual Emotion Analysis](https://mlsanity.com/paper/2109.01812/) (2021)
- [AngryBERT: Joint Learning Target and Emotion for Hate Speech Detection](https://mlsanity.com/paper/2103.11800/) (2021)
- [VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning](https://mlsanity.com/paper/2404.07078/) (2024)
- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [The Good, The Bad, and Why: Unveiling Emotions in Generative AI](https://mlsanity.com/paper/2312.11111/) (2023)
- [Do Moral Judgment and Reasoning Capability of LLMs Change with Language? A Study using the Multilingual Defining Issues Test](https://mlsanity.com/paper/2402.02135/) (2024)
- [Socratis: Are large multimodal models emotionally aware?](https://mlsanity.com/paper/2308.16741/) (2023)

## Built on by (most cited first, 19 of 19)

- [EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis](https://mlsanity.com/paper/2511.12554/) (2025)
- [VIBE: Can a VLM Read the Room?](https://mlsanity.com/paper/2506.11162/) (2025)
- [AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis](https://mlsanity.com/paper/2604.05900/) (2026)
- [Social Human Robot Embodied Conversation (SHREC) Dataset: Benchmarking Foundational Models' Social Reasoning](https://mlsanity.com/paper/2504.13898/) (2025)
- [Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models](https://mlsanity.com/paper/2509.19595/) (2025)
- [EmoGist: Efficient In-Context Learning for Visual Emotion Understanding](https://mlsanity.com/paper/2505.14660/) (2025)
- [Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?](https://mlsanity.com/paper/2512.17394/) (2025)
- [Do Vision Language Models Understand Human Engagement in Games?](https://mlsanity.com/paper/2603.18480/) (2026)
- [Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Norms](https://mlsanity.com/paper/2604.16757/) (2026)
- [ESC: Emotional Self-Correction for Reliable Vision-Language Models](https://mlsanity.com/paper/2607.02089/) (2026)
- [Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models](https://mlsanity.com/paper/2602.00123/) (2026)
- [AstroVLM: Expert Multi-agent Collaborative Reasoning for Astronomical Imaging Quality Diagnosis](https://mlsanity.com/paper/2604.16024/) (2026)
- [MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models](https://mlsanity.com/paper/2605.14635/) (2026)
- [VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2605.18547/) (2026)
- [Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?](https://mlsanity.com/paper/2607.12787/) (2026)
- [ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models](https://mlsanity.com/paper/2608.03358/) (2026)
- [FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows](https://mlsanity.com/paper/2608.21460/) (2026)
- [CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension](https://mlsanity.com/paper/2608.23172/) (2026)
- [Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities](https://mlsanity.com/paper/2609.07394/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (VLMs) have achieved unprecedented success in several objective multimodal reasoning tasks. However, to further enhance their capabilities of empathetic and effective communication with humans, improving how VLMs process and understand emotions is crucial. Despite significant research attention on improving affective understanding, there is a lack of detailed evaluations of VLMs for emotion-related tasks, which can potentially help inform downstream fine-tuning efforts. In this work, we present the first comprehensive evaluation of VLMs for recognizing evoked emotions from images. We create a benchmark for the task of evoked emotion recognition and study the performance of VLMs for this task, from perspectives of correctness and robustness. Through several experiments, we demonstrate important factors that emotion recognition performance depends on, and also characterize the various errors made by VLMs in the process. Finally, we pinpoint potential causes for errors through a human evaluation study. We use our experimental results to inform recommendations for the future of emotion research in the context of VLMs.