# FaceLLM: A Multimodal Large Language Model for Face Understanding

arXiv: https://arxiv.org/abs/2507.10300 · HTML: https://mlsanity.com/paper/2507.10300/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 72 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 23 of 23)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders](https://mlsanity.com/paper/2408.15998/) (2024)
- [HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding](https://mlsanity.com/paper/2501.15111/) (2025)
- [SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Models](https://mlsanity.com/paper/2402.04178/) (2024)
- [How Good is ChatGPT at Face Biometrics? A First Look into Recognition, Soft Biometrics, and Explainability](https://mlsanity.com/paper/2401.13641/) (2024)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [FaceXBench: Evaluating Multimodal LLMs on Face Understanding](https://mlsanity.com/paper/2501.10360/) (2025)
- [ChatGPT and biometrics: an assessment of face recognition, gender detection, and age estimation capabilities](https://mlsanity.com/paper/2403.02965/) (2024)
- [Robust Light-Weight Facial Affective Behavior Recognition with CLIP](https://mlsanity.com/paper/2403.09915/) (2024)
- [Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants](https://mlsanity.com/paper/2501.01243/) (2025)
- [Exploring ChatGPT for Face Presentation Attack Detection in Zero and Few-Shot in-Context Learning](https://mlsanity.com/paper/2501.08799/) (2025)
- [FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs](https://mlsanity.com/paper/2503.21457/) (2025)
- [ComFace: Facial Representation Learning with Synthetic Data for Comparing Faces](https://mlsanity.com/paper/2405.16016/) (2024)

## Built on by (most cited first, 15 of 15)

- [Benchmarking Multimodal Large Language Models for Face Recognition](https://mlsanity.com/paper/2510.14866/) (2025)
- [Evaluating Multimodal Large Language Models for Heterogeneous Face Recognition](https://mlsanity.com/paper/2601.15406/) (2026)
- [MLLM-based Textual Explanations for Face Comparison](https://mlsanity.com/paper/2603.16629/) (2026)
- [FPBench: A Comprehensive Benchmark of Multimodal Large Language Models for Fingerprint Analysis](https://mlsanity.com/paper/2512.18073/) (2025)
- [Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification](https://mlsanity.com/paper/2603.25613/) (2026)
- [Demographic and Linguistic Bias Evaluation in Omnimodal Language Models](https://mlsanity.com/paper/2604.10014/) (2026)
- [SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision](https://mlsanity.com/paper/2605.06179/) (2026)
- [Employing Vision-Language Models for Face Image Quality Assessment](https://mlsanity.com/paper/2605.17489/) (2026)
- [EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment](https://mlsanity.com/paper/2608.21486/) (2026)
- [Vision Language Model Fusion for Explainable Face Recognition](https://mlsanity.com/paper/2608.24430/) (2026)
- [Foundation and Multimodal Large Language Models for Face Presentation and Morph Attack Detection](https://mlsanity.com/paper/2608.29802/) (2026)
- [A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources](https://mlsanity.com/paper/2609.13255/) (2026)
- [Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition](https://mlsanity.com/paper/2609.21879/) (2026)
- [DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding](https://mlsanity.com/paper/2610.17015/) (2026)
- [When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs](https://mlsanity.com/paper/2610.17791/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have shown remarkable performance in vision-language tasks. However, existing MLLMs are primarily trained on generic datasets, limiting their ability to reason on domain-specific visual cues such as those in facial images. In particular, tasks that require detailed understanding of facial structure, expression, emotion, and demographic features remain underexplored by MLLMs due to the lack of large-scale annotated face image-text datasets. In this work, we introduce FaceLLM, a multimodal large language model trained specifically for facial image understanding. To construct the training data, we propose a novel weakly supervised pipeline that uses ChatGPT with attribute-aware prompts to generate high-quality question-answer pairs based on images from the FairFace dataset. The resulting corpus, called FairFaceGPT, covers a diverse set of attributes including expression, pose, skin texture, and forensic information. Our experiments demonstrate that FaceLLM improves the performance of MLLMs on various face-centric tasks and achieves state-of-the-art performance. This work highlights the potential of synthetic supervision via language models for building domain-specialized MLLMs, and sets a precedent for trustworthy, human-centric multimodal AI systems. FairFaceGPT dataset and pretrained FaceLLM models are publicly available in the project page.