mlsanityData to 14 Oct 2026

Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognition

arXiv 2506.19079 ↗First posted 23 Jun 2025cs.CV cs.AI cs.HCcited by 1

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Foundation Models (FMs) are rapidly transforming Affective Computing (AC), with Vision Language Models (VLMs) now capable of recognising emotions in zero shot settings. This paper probes a critical but underexplored question: what visual cues do these models rely on to infer affect, and are these cues psychologically grounded or superficially learnt? We benchmark varying scale VLMs on a teeth annotated subset of AffectNet dataset and find consistent performance shifts depending on the presence of visible teeth. Through structured introspection of, the best-performing model, i.e., GPT-4o, we show that facial attributes like eyebrow position drive much of its affective reasoning, revealing a high degree of internal consistency in its valence-arousal predictions. These patterns highlight the emergent nature of FMs behaviour, but also reveal risks: shortcut learning, bias, and fairness issues especially in sensitive domains like mental health and education.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 65 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

33 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Compound Expression Recognition via Large Vision-Language Models14 Mar
  2. 2024Contextual Emotion Recognition using Large Vision Language Models14 May

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  10. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  11. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 492
  12. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
19 more
  1. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  2. 2023Large Language Models for Robotics: A Surveycited by 133
  3. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  5. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 62
  6. 2021Understanding and Mitigating Annotation Bias in Facial Expression Recognitioncited by 50
  7. 2021Facial Expression Recognition with Visual Transformers and Attentional Selective Fusioncited by 49
  8. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  9. 2024Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigationcited by 25
  10. 2024The Clever Hans Effect in Unsupervised Learningcited by 24
  11. 2020The Clever Hans Effect in Anomaly Detectioncited by 13
  12. 2024Affective Computing Has Changed: The Foundation Model Disruptioncited by 7
  13. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  14. 2025Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Modelscited by 6
  15. 2020Generative Adversarial Networks in Human Emotion Synthesis:A Reviewcited by 4
  16. 2024Unsupervised Model Diagnosiscited by 4
  17. 2024Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformationcited by 3
  18. 2025Echo-Teddy: Preliminary Design and Development of Large Language Model-based Social Robot for Autistic Studentscited by 2
  19. 2024AI-Driven Early Mental Health Screening: Analyzing Selfies of Pregnant Womencited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Whose Face Is It Anyway? A Multi-Model Audit of Facial Affect Recognition on Children, and Why the Gap Is the Head, Not the Features
  2. 2025Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis
  3. 2025Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values
  4. 2026Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
  5. 2026Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?
  6. 2025EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition
  7. 2025FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning
  8. 2026The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception