# An Evaluation of a Visual Question Answering Strategy for Zero-shot Facial Expression Recognition in Still Images

arXiv: https://arxiv.org/abs/2504.21309 · HTML: https://mlsanity.com/paper/2504.21309/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions](https://mlsanity.com/paper/2401.02460/) (2024)
- [EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition](https://mlsanity.com/paper/2310.16640/) (2023)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion](https://mlsanity.com/paper/2412.04424/) (2024)
- [A Survey on Facial Expression Recognition of Static and Dynamic Emotions](https://mlsanity.com/paper/2408.15777/) (2024)
- [Visual Question Answering: which investigated applications?](https://mlsanity.com/paper/2103.02937/) (2021)
- [ResEmoteNet: Bridging Accuracy and Loss Reduction in Facial Emotion Recognition](https://mlsanity.com/paper/2409.10545/) (2024)

## Abstract (arXiv)

Facial expression recognition (FER) is a key research area in computer vision and human-computer interaction. Despite recent advances in deep learning, challenges persist, especially in generalizing to new scenarios. In fact, zero-shot FER significantly reduces the performance of state-of-the-art FER models. To address this problem, the community has recently started to explore the integration of knowledge from Large Language Models for visual tasks. In this work, we evaluate a broad collection of locally executed Visual Language Models (VLMs), avoiding the lack of task-specific knowledge by adopting a Visual Question Answering strategy. We compare the proposed pipeline with state-of-the-art FER models, both integrating and excluding VLMs, evaluating well-known FER benchmarks: AffectNet, FERPlus, and RAF-DB. The results show excellent performance for some VLMs in zero-shot FER scenarios, indicating the need for further exploration to improve FER generalization.