# From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models

arXiv: https://arxiv.org/abs/2408.14842 · HTML: https://mlsanity.com/paper/2408.14842/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Builds on (most cited first, 9 of 9)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Uncovering the Bias in Facial Expressions](https://mlsanity.com/paper/2011.11311/) (2020)
- [Addressing Racial Bias in Facial Emotion Recognition](https://mlsanity.com/paper/2308.04674/) (2023)
- [EmoFit: Affect Monitoring System for Sedentary Jobs](https://mlsanity.com/paper/1607.01077/) (2016)

## Built on by (most cited first, 1 of 1)

- [Modeling Cultural Bias in Facial Expression Recognition with Adaptive Agents](https://mlsanity.com/paper/2510.13557/) (2025)

## Abstract (arXiv)

This study addresses the racial biases in facial expression recognition (FER) systems within Large Multimodal Foundation Models (LMFMs). Despite advances in deep learning and the availability of diverse datasets, FER systems often exhibit higher error rates for individuals with darker skin tones. Existing research predominantly focuses on traditional FER models (CNNs, RNNs, ViTs), leaving a gap in understanding racial biases in LMFMs. We benchmark four leading LMFMs: GPT-4o, PaliGemma, Gemini, and CLIP to assess their performance in facial emotion detection across different racial demographics. A linear classifier trained on CLIP embeddings obtains accuracies of 95.9% for RADIATE, 90.3% for Tarr, and 99.5% for Chicago Face. Furthermore, we identify that Anger is misclassified as Disgust 2.1 times more often in Black Females than White Females. This study highlights the need for fairer FER systems and establishes a foundation for developing unbiased, accurate FER technologies. Visit https://kvjvhub.github.io/FERRacialBias/ for further information regarding the biases within facial expression recognition.