# A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models

arXiv: https://arxiv.org/abs/2505.20512 · HTML: https://mlsanity.com/paper/2505.20512/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 78 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning](https://mlsanity.com/paper/1911.10692/) (2019)
- [Fair Attribute Classification through Latent Space De-biasing](https://mlsanity.com/paper/2012.01469/) (2020)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Fair Contrastive Learning for Facial Attribute Classification](https://mlsanity.com/paper/2203.16209/) (2022)
- [Mitigating Face Recognition Bias via Group Adaptive Classifier](https://mlsanity.com/paper/2006.07576/) (2020)
- [Directional Bias Amplification](https://mlsanity.com/paper/2102.12594/) (2021)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias](https://mlsanity.com/paper/2212.11261/) (2022)
- [Consistent Instance False Positive Improves Fairness in Face Recognition](https://mlsanity.com/paper/2106.05519/) (2021)
- [American == White in Multimodal Language-and-Image AI](https://mlsanity.com/paper/2207.00691/) (2022)
- [Overwriting Pretrained Bias with Finetuning Data](https://mlsanity.com/paper/2303.06167/) (2023)
- [FairGRAPE: Fairness-aware GRAdient Pruning mEthod for Face Attribute Classification](https://mlsanity.com/paper/2207.10888/) (2022)
- [Responsible AI: Gender bias assessment in emotion recognition](https://mlsanity.com/paper/2103.11436/) (2021)
- [Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition](https://mlsanity.com/paper/2103.08637/) (2021)
- [Anatomizing Bias in Facial Analysis](https://mlsanity.com/paper/2112.06522/) (2021)
- [A study on the distribution of social biases in self-supervised learning visual models](https://mlsanity.com/paper/2203.01854/) (2022)

## Abstract (arXiv)

Recent studies on fairness have shown that Facial Expression Recognition (FER) models exhibit biases toward certain visually perceived demographic groups. However, the limited availability of human-annotated demographic labels in public FER datasets has constrained the scope of such bias analysis. To overcome this limitation, some prior works have resorted to pseudo-demographic labels, which may distort bias evaluation results. Alternatively, in this paper, we propose a feature-level bias evaluation framework for evaluating demographic biases in FER models under the setting where demographic labels are unavailable in the test set. Extensive experiments demonstrate that our method more effectively evaluates demographic biases compared to existing approaches that rely on pseudo-demographic labels. Furthermore, we observe that many existing studies do not include statistical testing in their bias evaluations, raising concerns that some reported biases may not be statistically significant but rather due to randomness. To address this issue, we introduce a plug-and-play statistical module to ensure the statistical significance of biased evaluation results. A comprehensive bias analysis based on the proposed module is then conducted across three sensitive attributes (age, gender, and race), seven facial expressions, and multiple network architectures on a large-scale dataset, revealing the prominent demographic biases in FER and providing insights on selecting a fairer network architecture.