mlsanityData to 7 Oct 2026

GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition

arXiv 2312.04293 ↗First posted 7 Dec 2023cs.CV cs.MMcited by 38

Not on the map. Why?

Abstract (arXiv)

Recently, GPT-4 with Vision (GPT-4V) has demonstrated remarkable visual capabilities across various tasks, but its performance in emotion recognition has not been fully evaluated. To bridge this gap, we present the quantitative evaluation results of GPT-4V on 21 benchmark datasets covering 6 tasks: visual sentiment analysis, tweet sentiment analysis, micro-expression recognition, facial emotion recognition, dynamic facial emotion recognition, and multimodal emotion recognition. This paper collectively refers to these tasks as ``Generalized Emotion Recognition (GER)''. Through experimental analysis, we observe that GPT-4V exhibits strong visual understanding capabilities in GER tasks. Meanwhile, GPT-4V shows the ability to integrate multimodal clues and exploit temporal information, which is also critical for emotion recognition. However, it's worth noting that GPT-4V is primarily designed for general domains and cannot recognize micro-expressions that require specialized knowledge. To the best of our knowledge, this paper provides the first quantitative assessment of GPT-4V for GER tasks. We have open-sourced the code and encourage subsequent researchers to broaden the evaluation scope by including more tasks and datasets. Our code and evaluation results are available at: https://github.com/zeroQiaoba/gpt4v-emotion.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 51 of its 112 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

51 of 112 references.

References · 51

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  12. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
38 more
  1. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  2. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  3. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  4. 2018Deep Facial Expression Recognition: A Surveycited by 269
  5. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  6. 2018Learning Factorized Multimodal Representationscited by 171
  7. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
  8. 2016Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distributioncited by 158
  9. 2020Suppressing Uncertainties for Large-Scale Facial Expression Recognitioncited by 157
  10. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  11. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  12. 2019Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognitioncited by 127
  13. 2021TransFER: Learning Relation-aware Facial Expression Representations with Transformerscited by 85
  14. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  15. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 81
  16. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  17. 2016Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmarkcited by 71
  18. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
  19. 2022Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognitioncited by 67
  20. 2019Frame attention networks for facial expression recognition in videoscited by 56
  21. 2021Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognitioncited by 55
  22. 2022POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognitioncited by 49
  23. 2015Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networkscited by 46
  24. 2021Deep Learning for Micro-expression Recognition: A Surveycited by 43
  25. 2023MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognitioncited by 42
  26. 2022Intensity-Aware Loss for Dynamic Facial Expression Recognition in the Wildcited by 41
  27. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  28. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 33
  29. 2022Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wildcited by 29
  30. 2019Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressionscited by 25
  31. 2023POSTER++: A simpler and stronger facial expression recognition networkcited by 25
  32. 2014DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networkscited by 24
  33. 2020MSAF: Multimodal Split Attention Fusioncited by 23
  34. 2021Learning to Amend Facial Expression Representation via De-albino and Affinitycited by 19
  35. 2023Explainable Multimodal Emotion Recognitioncited by 18
  36. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12
  37. 2021A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognitioncited by 11
  38. 2022Disentangling Identity and Pose for Facial Expression Recognitioncited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 20; 2026: 10.

38 papers.

Sort:

By area:

  1. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  2. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  3. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  4. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  5. 2026Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognitioncited by 11
  6. 2025VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injectioncited by 5
  7. 2024Multimodal Chain-of-Thought Reasoning via ChatGPT to Protect Children from Age-Inappropriate Appscited by 4
  8. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capabilitycited by 4
  9. 2025TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognitioncited by 4
  10. 2026UAVBench and UAVIT-1M: Benchmarking and Enhancing MLLMs for Low-Altitude UAV Vision-Language Understandingcited by 4
  11. 2025Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Chartscited by 3
  12. 2025Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyondcited by 3
26 more
  1. 2024Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognitioncited by 2
  2. 2025Can Large Language Models Capture Video Game Engagement?cited by 2
  3. 2025Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion Recognitioncited by 2
  4. 2025MELLM: A Flow-Guided Large Language Model for Micro-Expression Understandingcited by 2
  5. 2024ERIT Lightweight Multimodal Dataset for Elderly Emotion Recognition and Multimodal Fusion Evaluationcited by 1
  6. 2025Leveraging Large Language Models for Cost-Effective, Multilingual Depression Detection and Severity Assessmentcited by 1
  7. 2025Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMscited by 1
  8. 2025Synthetic History: Evaluating Visual Representations of the Past in Diffusion Modelscited by 1
  9. 2026XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoningcited by 1
  10. 2026EmoSpace: Immersive Affective Image Generation Guided by Fine-Grained Emotion Prototypescited by 1
  11. 2026Do Vision Language Models Understand Human Engagement in Games?cited by 1
  12. 2026EmoScene: A Dual-space Dataset for Controllable Affective Image Generationcited by 1
  13. 2024SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognitioncited by 0
  14. 2024Dynamic Adaptive Rank Space Exploration for Efficient Sentiment Analysis with Large Language Modelscited by 0
  15. 2025Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratingscited by 0
  16. 2025Contrastive Distillation of Emotion Knowledge from LLMs for Zero-Shot Emotion Recognitioncited by 0
  17. 2025Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generationcited by 0
  18. 2025Perspectives on How Sociology Can Advance Theorizing about Human-Chatbot Interaction and Developing Chatbots for Social Goodcited by 0
  19. 2025Salience Adjustment for Context-Based Emotion Recognitioncited by 0
  20. 2025Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captionscited by 0
  21. 2025Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognitioncited by 0
  22. 2025QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysiscited by 0
  23. 2026Towards LLM-centric Affective Visual Customization via Efficient and Precise Emotion Manipulatingcited by 0
  24. 2026Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challengecited by 0
  25. 2026Facial-Expression-Aware Prompting for Empathetic LLM Tutoringcited by 0
  26. 2026Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Modelscited by 0