# Addressing Racial Bias in Facial Emotion Recognition

arXiv: https://arxiv.org/abs/2308.04674 · HTML: https://mlsanity.com/paper/2308.04674/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [The Frontiers of Fairness in Machine Learning](https://mlsanity.com/paper/1810.08810/) (2018)
- [Fairness in Deep Learning: A Computational Perspective](https://mlsanity.com/paper/1908.08843/) (2019)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Responsible AI: Gender bias assessment in emotion recognition](https://mlsanity.com/paper/2103.11436/) (2021)
- [Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?](https://mlsanity.com/paper/1510.02969/) (2015)
- [Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey](https://mlsanity.com/paper/1911.05609/) (2019)
- [Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks](https://mlsanity.com/paper/1705.07871/) (2017)
- [Fairness and Missing Values](https://mlsanity.com/paper/1905.12728/) (2019)
- [Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study](https://mlsanity.com/paper/2012.08678/) (2020)
- [Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels](https://mlsanity.com/paper/2101.03477/) (2021)

## Built on by (most cited first, 4 of 4)

- [From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models](https://mlsanity.com/paper/2408.14842/) (2024)
- [TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone](https://mlsanity.com/paper/2505.20637/) (2025)
- [Modeling Cultural Bias in Facial Expression Recognition with Adaptive Agents](https://mlsanity.com/paper/2510.13557/) (2025)
- [A Hybrid Deep Learning Framework for Emotion Recognition in Children with Autism During NAO Robot-Mediated Interaction](https://mlsanity.com/paper/2512.12208/) (2025)

## Abstract (arXiv)

Fairness in deep learning models trained with high-dimensional inputs and subjective labels remains a complex and understudied area. Facial emotion recognition, a domain where datasets are often racially imbalanced, can lead to models that yield disparate outcomes across racial groups. This study focuses on analyzing racial bias by sub-sampling training sets with varied racial distributions and assessing test performance across these simulations. Our findings indicate that smaller datasets with posed faces improve on both fairness and performance metrics as the simulations approach racial balance. Notably, the F1-score increases by 27.2% points, and demographic parity increases by 15.7% points on average across the simulations. However, in larger datasets with greater facial variation, fairness metrics generally remain constant, suggesting that racial balance by itself is insufficient to achieve parity in test performance across different racial groups.