# Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models

arXiv: https://arxiv.org/abs/2502.11049 · HTML: https://mlsanity.com/paper/2502.11049/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 86 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Fairness And Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, And Mitigation Strategies](https://mlsanity.com/paper/2304.07683/) (2023)
- [The Conditional Entropy Bottleneck](https://mlsanity.com/paper/2002.05379/) (2020)
- [From Facial Expression Recognition to Interpersonal Relation Prediction](https://mlsanity.com/paper/1609.06426/) (2016)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition](https://mlsanity.com/paper/2204.04083/) (2022)
- [Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition](https://mlsanity.com/paper/2210.09943/) (2022)
- [A Deeper Look at Facial Expression Dataset Bias](https://mlsanity.com/paper/1904.11150/) (2019)
- [Potential sources of dataset bias complicate investigation of underdiagnosis by machine learning algorithms](https://mlsanity.com/paper/2201.07856/) (2022)
- [Anatomizing Bias in Facial Analysis](https://mlsanity.com/paper/2112.06522/) (2021)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition](https://mlsanity.com/paper/2303.15889/) (2023)
- [Uncovering the Bias in Facial Expressions](https://mlsanity.com/paper/2011.11311/) (2020)
- [Bias and Diversity in Synthetic-based Face Recognition](https://mlsanity.com/paper/2311.03970/) (2023)
- [Towards A Holistic View of Bias in Machine Learning: Bridging Algorithmic Fairness and Imbalanced Learning](https://mlsanity.com/paper/2207.06084/) (2022)

## Built on by (most cited first, 6 of 6)

- [Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognition](https://mlsanity.com/paper/2506.19079/) (2025)
- [Investigating Associational Biases in Inter-Model Communication of Large Generative Models](https://mlsanity.com/paper/2601.22093/) (2026)
- [Investigating Bias and Fairness in Appearance-based Gaze Estimation](https://mlsanity.com/paper/2604.10707/) (2026)
- [Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness](https://mlsanity.com/paper/2504.14882/) (2025)
- [TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone](https://mlsanity.com/paper/2505.20637/) (2025)
- [Architectural Bias in Face Presentation Attack Detection: A Comparative Study of Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2606.18510/) (2026)

## Abstract (arXiv)

Automated Facial Expression Recognition (FER), involves two critical aspects: data and model design. Both significantly influence bias and fairness in FER tasks. However, issues related to bias and fairness in FER datasets and models remain underexplored. This study investigates bias and fairness in FER datasets and models. The bias of four common in-the-wild FER datasets, including AffectNet, ExpW, Fer2013, and RAF-DB, is studied. Additionally, this research evaluates the bias and fairness of seven deep models, including three generic CNN models: MobileNet, ResNet, XceptionNet, as well as two popular Transformer-based models: ViT and CLIP, plus two FER-specific state-of-the-art models: POSTER and CEPrompt. Unlike prior studies that examine only limited aspects of bias, our work introduces a unified evaluation framework for FER that integrates five existing and two newly proposed dataset metrics with four fairness criteria for model analysis. We further introduce two new metrics, Conditional-Entropy Bias Index and Concentration Index, designed to quantify conditional dependencies and intra-group data imbalance that existing measures fail to capture. Our results show that all four datasets carry significant demographic bias, most notably in race, with AffectNet exhibiting the highest overall bias and Fer2013 the lowest. At the model level, we find that residual-based CNN architectures (ResNet and XceptionNet) exhibit the lowest overall bias, whereas Transformer-based models (ViT and CLIP) exhibit the highest, despite often achieving comparable or superior accuracy. These findings demonstrate that high predictive accuracy does not guarantee fairness, and that dataset-level and model-level bias must be addressed jointly rather than in isolation.