# Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

arXiv: https://arxiv.org/abs/2303.15889 · HTML: https://mlsanity.com/paper/2303.15889/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 111 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Algorithmic Fairness](https://mlsanity.com/paper/2001.09784/) (2020)
- [REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets](https://mlsanity.com/paper/2004.07999/) (2020)
- [Understanding and Evaluating Racial Biases in Image Captioning](https://mlsanity.com/paper/2106.08503/) (2021)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [A Survey on Bias in Visual Datasets](https://mlsanity.com/paper/2107.07919/) (2021)
- [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021)
- [From Facial Expression Recognition to Interpersonal Relation Prediction](https://mlsanity.com/paper/1609.06426/) (2016)
- [Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets](https://mlsanity.com/paper/1905.01347/) (2019)
- [Responsible AI: Gender bias assessment in emotion recognition](https://mlsanity.com/paper/2103.11436/) (2021)
- [Max-Margin Object Detection](https://mlsanity.com/paper/1502.00046/) (2015)
- [Uncovering the Bias in Facial Expressions](https://mlsanity.com/paper/2011.11311/) (2020)
- ["Excavating AI" Re-excavated: Debunking a Fallacious Account of the JAFFE Dataset](https://mlsanity.com/paper/2107.13998/) (2021)
- [A novel database of Children's Spontaneous Facial Expressions (LIRIS-CSE)](https://mlsanity.com/paper/1812.01555/) (2018)
- [Assessing Demographic Bias Transfer from Dataset to Model: A Case Study in Facial Expression Recognition](https://mlsanity.com/paper/2205.10049/) (2022)
- [Comparing Human and Machine Bias in Face Recognition](https://mlsanity.com/paper/2110.08396/) (2021)

## Built on by (most cited first, 13 of 13)

- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [DSAP: Analyzing Bias Through Demographic Comparison of Datasets](https://mlsanity.com/paper/2312.14626/) (2023)
- [Less can be more: representational vs. stereotypical gender bias in facial expression recognition](https://mlsanity.com/paper/2406.17405/) (2024)
- [CausalAffect: Causal Discovery for Facial Affective Understanding](https://mlsanity.com/paper/2512.00456/) (2025)
- [Underrepresentation, Label Bias, and Proxies: Towards Data Bias Profiles for the EU AI Act and Beyond](https://mlsanity.com/paper/2507.08866/) (2025)
- [Addressing the ID-Matching Challenge in Long Video Captioning](https://mlsanity.com/paper/2510.06973/) (2025)
- [Biased Heritage: How Datasets Shape Models in Facial Expression Recognition](https://mlsanity.com/paper/2503.03446/) (2025)
- [A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models](https://mlsanity.com/paper/2505.20512/) (2025)
- [KG-FairDiff: Knowledge Graph-Guided Prompt Refinement for Demographically Fair Text-to-Image Generation](https://mlsanity.com/paper/2606.01282/) (2026)
- [Facial Affect Analysis for Service-Oriented Systems: Advances, Challenges, and Future Visions](https://mlsanity.com/paper/2606.15351/) (2026)
- [Unmasking LAION-5B: Age, Gender, Race, and Emotion Biases in Large-Scale Image Datasets](https://mlsanity.com/paper/2606.23204/) (2026)
- [AU-Guided Synthetic Video Generation for Micro-Expression Recognition](https://mlsanity.com/paper/2607.10860/) (2026)
- [When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs](https://mlsanity.com/paper/2610.17791/) (2026)

## Abstract (arXiv)

Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this paper, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models. The code is available at https://github.com/irisdominguez/dataset_bias_metrics.