mlsanityData to 7 Oct 2026

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

arXiv 2303.15889 ↗First posted 28 Mar 2023cs.CV cs.CYcited by 12

Not on the map. Why?

Abstract (arXiv)

Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this paper, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models. The code is available at https://github.com/irisdominguez/dataset_bias_metrics.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 33 of its 111 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

33 of 111 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  3. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  4. 2018Model Cards for Model Reportingcited by 1,068
  5. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  6. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  7. 2017Fairness in Criminal Justice Risk Assessments: The State of the Artcited by 315
  8. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  9. 2018Deep Facial Expression Recognition: A Surveycited by 269
  10. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  11. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  12. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
21 more
  1. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 192
  2. 2016Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distributioncited by 158
  3. 2020Algorithmic Fairnesscited by 143
  4. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  5. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  6. 2019Context-Aware Emotion Recognition Networkscited by 99
  7. 2020Investigating Bias and Fairness in Facial Expression Recognitioncited by 87
  8. 2021A Survey on Bias in Visual Datasetscited by 84
  9. 2021Towards Measuring Fairness in AI: the Casual Conversations Datasetcited by 80
  10. 2016From Facial Expression Recognition to Interpersonal Relation Predictioncited by 53
  11. 2019Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasetscited by 24
  12. 2021Responsible AI: Gender bias assessment in emotion recognitioncited by 22
  13. 2015Max-Margin Object Detectioncited by 15
  14. 2020Uncovering the Bias in Facial Expressionscited by 12
  15. 2021"Excavating AI" Re-excavated: Debunking a Fallacious Account of the JAFFE Datasetcited by 12
  16. 2018A novel database of Children's Spontaneous Facial Expressions (LIRIS-CSE)cited by 10
  17. 2022Assessing Demographic Bias Transfer from Dataset to Model: A Case Study in Facial Expression Recognitioncited by 9
  18. 2021Comparing Human and Machine Bias in Face Recognitioncited by 8
  19. 2022Gender Stereotyping Impact in Facial Expression Recognitioncited by 8
  20. 2021Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groupscited by 4
  21. 2023The Casual Conversations v2 Datasetcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 6; 2026: 4.

12 papers.

  1. 2025Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Modelscited by 6
  2. 2023DSAP: Analyzing Bias Through Demographic Comparison of Datasetscited by 5
  3. 2024Less can be more: representational vs. stereotypical gender bias in facial expression recognitioncited by 5
  4. 2025CausalAffect: Causal Discovery for Facial Affective Understandingcited by 3
  5. 2025Underrepresentation, Label Bias, and Proxies: Towards Data Bias Profiles for the EU AI Act and Beyondcited by 2
  6. 2025Biased Heritage: How Datasets Shape Models in Facial Expression Recognitioncited by 1
  7. 2025Addressing the ID-Matching Challenge in Long Video Captioningcited by 1
  8. 2025A Feature-level Bias Evaluation Framework for Facial Expression Recognition Modelscited by 0
  9. 2026KG-FairDiff: Knowledge Graph-Guided Prompt Refinement for Demographically Fair Text-to-Image Generationcited by 0
  10. 2026Facial Affect Analysis for Service-Oriented Systems: Advances, Challenges, and Future Visionscited by 0
  11. 2026Unmasking LAION-5B: Age, Gender, Race, and Emotion Biases in Large-Scale Image Datasetscited by 0
  12. 2026AU-Guided Synthetic Video Generation for Micro-Expression Recognitioncited by 0