mlsanityData to 7 Oct 2026

FaceXBench: Evaluating Multimodal LLMs on Face Understanding

arXiv 2501.10360 ↗First posted 17 Jan 2025cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) demonstrate impressive problem-solving abilities across a wide range of tasks and domains. However, their capacity for face understanding has not been systematically studied. To address this gap, we introduce FaceXBench, a comprehensive benchmark designed to evaluate MLLMs on complex face understanding tasks. FaceXBench includes 5,000 multimodal multiple-choice questions derived from 25 public datasets and a newly created dataset, FaceXAPI. These questions cover 14 tasks across 6 broad categories, assessing MLLMs' face understanding abilities in bias and fairness, face authentication, recognition, analysis, localization and tool retrieval. Using FaceXBench, we conduct an extensive evaluation of 26 open-source MLLMs alongside 2 proprietary models, revealing the unique challenges in complex face understanding tasks. We analyze the models across three evaluation settings: zero-shot, in-context task description, and chain-of-thought prompting. Our detailed analysis reveals that current MLLMs, including advanced models like GPT-4o, and GeminiPro 1.5, show significant room for improvement. We believe FaceXBench will be a crucial resource for developing MLLMs equipped to perform sophisticated face understanding. Code: https://github.com/Kartik-3004/facexbench

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 113 references that are in the map. First posted on arXiv 17 Jan 2025.

Builds on

86 of 113 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Face-MLLM: A Large Face Perception Model28 Oct

Other references · 85

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
38 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  14. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  16. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  17. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  18. 2023A Survey on Multimodal Large Language Modelscited by 898
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  20. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  21. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  22. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  23. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  24. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  25. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  26. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  27. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  28. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  29. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  30. 2021Multi-attentional Deepfake Detectioncited by 339
  31. 2024What matters when building vision-language models?cited by 321
  32. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  33. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  34. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  35. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  36. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  37. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  38. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 12.

23 papers.

Sort:

By area:

  1. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 18
  2. 2025FaceLLM: A Multimodal Large Language Model for Face Understandingcited by 13
  3. 2025Foundation versus Domain-specific Models: Performance Comparison, Fusion, and Explainability in Face Recognitioncited by 8
  4. 2025AvatarShield: Visual Reinforcement Learning for Human-Centric Synthetic Video Detectioncited by 6
  5. 2025Benchmarking Multimodal Large Language Models for Face Recognitioncited by 5
  6. 2026Evaluating Multimodal Large Language Models for Heterogeneous Face Recognitioncited by 3
  7. 2025FaceInsight: A Multimodal Large Language Model for Face Perceptioncited by 2
  8. 2026MLLM-based Textual Explanations for Face Comparisoncited by 2
  9. 2025M-PACE: Mother Child Framework for Multimodal Compliancecited by 1
  10. 2025FPBench: A Comprehensive Benchmark of Multimodal Large Language Models for Fingerprint Analysiscited by 1
  11. 2026Focal-RegionFace: Generating Fine-Grained Multi-attribute Descriptions for Arbitrarily Selected Face Focal Regionscited by 1
  12. 2026Emergent Morphing Attack Detection in Open Multi-modal Large Language Modelscited by 1
11 more
  1. 2026Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verificationcited by 1
  2. 2025In-context Learning of Vision Language Models for Detection of Physical and Digital Attacks against Face Recognition Systemscited by 0
  3. 2025Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captionscited by 0
  4. 2025Reliable and Reproducible Demographic Inference for Fairness in Face Analysiscited by 0
  5. 2026Demographic and Linguistic Bias Evaluation in Omnimodal Language Modelscited by 0
  6. 2026FaceMoE: Mixture of Experts for Low-Resolution Face Recognitioncited by 0
  7. 2026SLAPBench: Benchmarking Multimodal Large Language Models for Four-Finger SLAP Fingerprint Verificationcited by 0
  8. 20263D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanismcited by 0
  9. 2026Vision Language Model Fusion for Explainable Face Recognitioncited by 0
  10. 2026Beauty is in the AI of the beholder: MLLMs systematically overrate facial attractivenesscited by 0
  11. 2026Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs
  2. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  3. 2025Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants
  4. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  5. 2024A Survey on Benchmarks of Multimodal Large Language Models
  6. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  7. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
  8. 2025MANBench: Is Your Multimodal Model Smarter than Human?