# On the Illusion of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes

arXiv: https://arxiv.org/abs/2501.12020 · HTML: https://mlsanity.com/paper/2501.12020/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 64 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 24 of 24)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Analysis of Gender Inequality In Face Recognition Accuracy](https://mlsanity.com/paper/2002.00065/) (2020)
- [Fairness in Biometrics: a figure of merit to assess biometric verification systems](https://mlsanity.com/paper/2011.02395/) (2020)
- [PASS: Protected Attribute Suppression System for Mitigating Bias in Face Recognition](https://mlsanity.com/paper/2108.03764/) (2021)
- [How Does Gender Balance In Training Data Affect Face Recognition Accuracy?](https://mlsanity.com/paper/2002.02934/) (2020)
- [MAAD-Face: A Massively Annotated Attribute Dataset for Face Images](https://mlsanity.com/paper/2012.01030/) (2020)
- [Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphology](https://mlsanity.com/paper/2112.14656/) (2021)
- [Post-Comparison Mitigation of Demographic Bias in Face Recognition Using Fair Score Normalization](https://mlsanity.com/paper/2002.03592/) (2020)
- [Face Recognition Accuracy Across Demographics: Shining a Light Into the Problem](https://mlsanity.com/paper/2206.01881/) (2022)
- [Is Face Recognition Sexist? No, Gendered Hairstyles and Biology Are](https://mlsanity.com/paper/2008.06989/) (2020)
- [Beyond Identity: What Information Is Stored in Biometric Face Templates?](https://mlsanity.com/paper/2009.09918/) (2020)
- [Evaluating Proposed Fairness Models for Face Recognition Algorithms](https://mlsanity.com/paper/2203.05051/) (2022)
- [Stacked Dense U-Nets with Dual Transformers for Robust Face Alignment](https://mlsanity.com/paper/1812.01936/) (2018)
- [The Gender Gap in Face Recognition Accuracy Is a Hairy Problem](https://mlsanity.com/paper/2206.04867/) (2022)
- [Logical Consistency and Greater Descriptive Power for Facial Hair Attribute Learning](https://mlsanity.com/paper/2302.11102/) (2023)
- [What Should Be Balanced in a "Balanced" Face Recognition Dataset?](https://mlsanity.com/paper/2304.09818/) (2023)
- [Distill and De-bias: Mitigating Bias in Face Verification using Knowledge Distillation](https://mlsanity.com/paper/2112.09786/) (2021)
- [Beard Segmentation and Recognition Bias](https://mlsanity.com/paper/2308.15740/) (2023)
- [On the Reliability of Biometric Datasets: How Much Test Data Ensures Reliability?](https://mlsanity.com/paper/2501.06504/) (2025)

## Built on by (most cited first, 2 of 2)

- [Review of Demographic Fairness in Face Recognition](https://mlsanity.com/paper/2502.02309/) (2025)
- [A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks](https://mlsanity.com/paper/2505.19920/) (2025)

## Abstract (arXiv)

Face recognition systems (FRS) exhibit significant accuracy differences based on the user's gender. Since such a gender gap reduces the trustworthiness of FRS, more recent efforts have tried to find the causes. However, these studies make use of manually selected, correlated, and small-sized sets of facial features to support their claims. In this work, we analyze gender bias in face recognition by successfully extending the search domain to decorrelated combinations of 40 non-demographic facial characteristics. First, we introduce a toolchain to effectively decorrelate and aggregate facial attributes to enable a less-biased gender analysis on large-scale data. Second, we tailor two specialized metrics to quantify the effect of facial attributes on absolute and relative fairness. Based on these grounds, we thirdly present a novel unsupervised joint investigation framework capable of identifying attribute combinations leading to vanishing bias when used as filter predicates for balanced testing datasets. Experiments show the gender gap vanishing when images of male and female subjects share specific attributes, clearly indicating that the disparate performance is not a question of biology but of the social definition of appearance. These findings could reshape our understanding of fairness in face biometrics and provide insights into FRS, helping to address gender bias issues.