mlsanityData to 7 Oct 2026

GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing

arXiv 2403.05916 ↗First posted 9 Mar 2024cs.CV cs.AIcited by 22

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Multimodal large language models (MLLMs) are designed to process and integrate information from multiple sources, such as text, speech, images, and videos. Despite its success in language understanding, it is critical to evaluate the performance of downstream tasks for better human-centric applications. This paper assesses the application of MLLMs with 5 crucial abilities for affective computing, spanning from visual affective tasks and reasoning tasks. The results show that gpt has high accuracy in facial action unit recognition and micro-expression detection while its general facial expression recognition performance is not accurate. We also highlight the challenges of achieving fine-grained micro-expression recognition and the potential for further study and demonstrate the versatility and potential of gpt for handling advanced tasks in emotion recognition and related fields by integrating with task-related agents for more complex tasks, such as heart rate estimation through signal processing. In conclusion, this paper provides valuable insights into the potential applications and challenges of MLLMs in human-centric computing. Our interesting examples are at https://github.com/EnVision-Research/GPT4Affectivity.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 51 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

27 of 51 references.

References · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  7. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  8. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  9. 2018DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networkscited by 153
  10. 2021PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformercited by 118
  11. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  12. 2019RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representationcited by 104
15 more
  1. 2022Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognitioncited by 81
  2. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  3. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  4. 2022A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advancescited by 59
  5. 2018Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignmentcited by 55
  6. 2019Semantic Relationships Guided Representation Learning for Facial Action Unit Recognitioncited by 53
  7. 2018Facial Action Unit Detection Using Attention and Relation Learningcited by 46
  8. 2023PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformercited by 46
  9. 2024Video Anomaly Detection and Explanation via Large Language Modelscited by 45
  10. 2018Deep Structure Inference Network for Facial Action Unit Recognitioncited by 38
  11. 2023rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurementcited by 37
  12. 2023Neuron Structure Modeling for Generalizable Remote Physiological Measurementcited by 29
  13. 2021Self-supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Losscited by 23
  14. 2023Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learningcited by 14
  15. 2023Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detectioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024Hawk: Learning to Understand Open-World Video Anomaliescited by 47
  2. 2024A Survey of Useful LLM Evaluationcited by 28
  3. 2025Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Modelscited by 27
  4. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  5. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  6. 2024Representation Learning and Identity Adversarial Training for Facial Behavior Understandingcited by 12
  7. 2024PhysMLE: Generalizable and Priors-Inclusive Multi-task Remote Physiological Measurementcited by 9
  8. 2025CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generationcited by 9
  9. 2024Language-Driven Interactive Shadow Detectioncited by 7
  10. 2025CartoAgent: a multimodal large language model-powered multi-agent cartographic framework for map style transfer and evaluationcited by 4
  11. 2025Align the GAP: Prior-based Unified Multi-Task Remote Physiological Measurement Framework For Domain Generalization and Personalizationcited by 4
  12. 2025Sage Deer: A Super-Aligned Driving Generalist Is Your Copilotcited by 2
10 more
  1. 2025Not Only Consistency: Enhance Test-Time Adaptation with Spatio-temporal Inconsistency for Remote Physiological Measurementcited by 2
  2. 2024MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective Computingcited by 1
  3. 2025VAEER: Visual Attention-Inspired Emotion Elicitation Reasoningcited by 1
  4. 2026Do Vision Language Models Understand Human Engagement in Games?cited by 1
  5. 2026MILD: Mediator Agent System with Bidirectional Perception and Multi-Layered Alignment for Human-Vehicle Collaborationcited by 1
  6. 2024Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweightercited by 0
  7. 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Modelcited by 0
  8. 2025Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognitioncited by 0
  9. 2025FedHUG: Federated Heterogeneous Unsupervised Generalization for Remote Physiological Measurementscited by 0
  10. 2026Zero-MELO: Test-Time Evidence Calibration with Multimodal LLMs for Zero-Shot Micro-Gesture Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
  2. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  3. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
  4. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
  5. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning
  6. 2025Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
  7. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
  8. 2025Evaluating Vision-Language Models for Emotion Recognition