mlsanityData to 14 Oct 2026

Evaluating Open-Source Vision Language Models for Facial Emotion Recognition against Traditional Deep Learning Models

arXiv 2508.13524 ↗First posted 19 Aug 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Facial Emotion Recognition (FER) is crucial for applications such as human-computer interaction and mental health diagnostics. This study presents the first empirical comparison of open-source Vision-Language Models (VLMs), including Phi-3.5 Vision and CLIP, against traditional deep learning models VGG19, ResNet-50, and EfficientNet-B0 on the challenging FER-2013 dataset, which contains 35,887 low-resolution grayscale images across seven emotion classes. To address the mismatch between VLM training assumptions and the noisy nature of FER data, we introduce a novel pipeline that integrates GFPGAN-based image restoration with FER evaluation. Results show that traditional models, particularly EfficientNet-B0 (86.44%) and ResNet-50 (85.72%), significantly outperform VLMs like CLIP (64.07%) and Phi-3.5 Vision (51.66%), highlighting the limitations of VLMs in low-quality visual tasks. In addition to performance evaluation using precision, recall, F1-score, and accuracy, we provide a detailed computational cost analysis covering preprocessing, training, inference, and evaluation phases, offering practical insights for deployment. This work underscores the need for adapting VLMs to noisy environments and provides a reproducible benchmark for future research in emotion recognition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 39 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

24 of 39 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2021Scaling Vision Transformerscited by 999
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
12 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  2. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 492
  3. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  4. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 349
  5. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  6. 2018Deep Facial Expression Recognition: A Surveycited by 270
  7. 2015Object detection via a multi-region & semantic segmentation-aware CNN modelcited by 47
  8. 2021Facial Emotion Recognition: State of the Art Performance on FER2013cited by 20
  9. 2024Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategiescited by 9
  10. 2016Facial Expression Recognition Using a Hybrid CNN-SIFT Aggregatorcited by 4
  11. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMscited by 4
  12. 2024Towards Context-Rich Automated Biodiversity Assessments: Deriving AI-Powered Insights from Camera Trap Datacited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Vision-Language Models for Emotion Recognition
  2. 2026Why Do Vision Language Models Struggle To Recognize Human Emotions?
  3. 2023EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition
  4. 2025An Evaluation of a Visual Question Answering Strategy for Zero-shot Facial Expression Recognition in Still Images
  5. 2025Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond
  6. 2026Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
  7. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning
  8. 2024Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model