# Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age

arXiv: https://arxiv.org/abs/2410.24148 · HTML: https://mlsanity.com/paper/2410.24148/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 75 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra](https://mlsanity.com/paper/2404.03647/) (2024)
- [POSTER++: A simpler and stronger facial expression recognition network](https://mlsanity.com/paper/2301.12149/) (2023)
- [Understanding bias in facial recognition technologies](https://mlsanity.com/paper/2010.07023/) (2020)
- [AI-generated faces influence gender stereotypes and racial homogenization](https://mlsanity.com/paper/2402.01002/) (2024)
- [Representation Learning and Identity Adversarial Training for Facial Behavior Understanding](https://mlsanity.com/paper/2407.11243/) (2024)
- [Bias and Fairness on Multimodal Emotion Detection Algorithms](https://mlsanity.com/paper/2205.08383/) (2022)
- [What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models](https://mlsanity.com/paper/2405.15668/) (2024)
- [Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers](https://mlsanity.com/paper/2207.11081/) (2022)
- [Harnessing Large Vision and Language Models in Agriculture: A Review](https://mlsanity.com/paper/2407.19679/) (2024)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Heterogeneous Face Attribute Estimation](https://mlsanity.com/paper/2403.00561/) (2024)

## Built on by (most cited first, 4 of 4)

- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [Human Re-ID Meets LVLMs: What can we expect?](https://mlsanity.com/paper/2501.18698/) (2025)
- [VLAgeBench: Benchmarking Large Vision-Language Models for Zero-Shot Human Age Estimation](https://mlsanity.com/paper/2603.26015/) (2026)
- [3D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanism](https://mlsanity.com/paper/2607.16280/) (2026)

## Abstract (arXiv)

Technologies for recognizing facial attributes like race, gender, age, and emotion have several applications, such as surveillance, advertising content, sentiment analysis, and the study of demographic trends and social behaviors. Analyzing demographic characteristics based on images and analyzing facial expressions have several challenges due to the complexity of humans' facial attributes. Traditional approaches have employed CNNs and various other deep learning techniques, trained on extensive collections of labeled images. While these methods demonstrated effective performance, there remains potential for further enhancements. In this paper, we propose to utilize vision language models (VLMs) such as generative pre-trained transformer (GPT), GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 to recognize facial attributes such as race, gender, age, and emotion from images with human faces. Various datasets like FairFace, AffectNet, and UTKFace have been utilized to evaluate the solutions. The results show that VLMs are competitive if not superior to traditional techniques. Additionally, we propose "FaceScanPaliGemma"--a fine-tuned PaliGemma model--for race, gender, age, and emotion recognition. The results show an accuracy of 81.1%, 95.8%, 80%, and 59.4% for race, gender, age group, and emotion classification, respectively, outperforming pre-trained version of PaliGemma, other VLMs, and SotA methods. Finally, we propose "FaceScanGPT", which is a GPT-4o model to recognize the above attributes when several individuals are present in the image using a prompt engineered for a person with specific facial and/or physical attributes. The results underscore the superior multitasking capability of FaceScanGPT to detect the individual's attributes like hair cut, clothing color, postures, etc., using only a prompt to drive the detection and recognition tasks.