# Uncovering the Bias in Facial Expressions

arXiv: https://arxiv.org/abs/2011.11311 · HTML: https://mlsanity.com/paper/2011.11311/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023)
- [CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?](https://mlsanity.com/paper/2403.04547/) (2024)
- [Anatomizing Bias in Facial Analysis](https://mlsanity.com/paper/2112.06522/) (2021)
- [Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition](https://mlsanity.com/paper/2303.15889/) (2023)
- [A Reduction to Binary Approach for Debiasing Multiclass Datasets](https://mlsanity.com/paper/2205.15860/) (2022)
- [Gender Stereotyping Impact in Facial Expression Recognition](https://mlsanity.com/paper/2210.05332/) (2022)
- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [Less can be more: representational vs. stereotypical gender bias in facial expression recognition](https://mlsanity.com/paper/2406.17405/) (2024)
- [FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations](https://mlsanity.com/paper/2412.07313/) (2024)
- [From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models](https://mlsanity.com/paper/2408.14842/) (2024)
- [Biased Heritage: How Datasets Shape Models in Facial Expression Recognition](https://mlsanity.com/paper/2503.03446/) (2025)

## Abstract (arXiv)

Over the past decades the machine and deep learning community has celebrated great achievements in challenging tasks such as image classification. The deep architecture of artificial neural networks together with the plenitude of available data makes it possible to describe highly complex relations. Yet, it is still impossible to fully capture what the deep learning model has learned and to verify that it operates fairly and without creating bias, especially in critical tasks, for instance those arising in the medical field. One example for such a task is the detection of distinct facial expressions, called Action Units, in facial images. Considering this specific task, our research aims to provide transparency regarding bias, specifically in relation to gender and skin color. We train a neural network for Action Unit classification and analyze its performance quantitatively based on its accuracy and qualitatively based on heatmaps. A structured review of our results indicates that we are able to detect bias. Even though we cannot conclude from our results that lower classification performance emerged solely from gender and skin color bias, these biases must be addressed, which is why we end by giving suggestions on how the detected bias can be avoided.