# What Should Be Balanced in a "Balanced" Face Recognition Dataset?

arXiv: https://arxiv.org/abs/2304.09818 · HTML: https://mlsanity.com/paper/2304.09818/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015)
- [Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning](https://mlsanity.com/paper/1609.06570/) (2016)
- [Partial FC: Training 10 Million Identities on a Single Machine](https://mlsanity.com/paper/2010.05222/) (2020)
- [Demographic Bias in Biometrics: A Survey on an Emerging Challenge](https://mlsanity.com/paper/2003.02488/) (2020)
- [FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age](https://mlsanity.com/paper/1908.04913/) (2019)
- [Face Recognition: Too Bias, or Not Too Bias?](https://mlsanity.com/paper/2002.06483/) (2020)
- [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021)
- [img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation](https://mlsanity.com/paper/2012.07791/) (2020)
- [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.07698/) (2018)
- [Meta Balanced Network for Fair Face Recognition](https://mlsanity.com/paper/2205.06548/) (2022)
- [FaceX-Zoo: A PyTorch Toolbox for Face Recognition](https://mlsanity.com/paper/2101.04407/) (2021)
- [Analysis of Gender Inequality In Face Recognition Accuracy](https://mlsanity.com/paper/2002.00065/) (2020)
- [Biometric Quality: Review and Application to Face Recognition with FaceQnet](https://mlsanity.com/paper/2006.03298/) (2020)
- [Characterizing the Variability in Face Recognition Accuracy Relative to Race](https://mlsanity.com/paper/1904.07325/) (2019)
- [How Does Gender Balance In Training Data Affect Face Recognition Accuracy?](https://mlsanity.com/paper/2002.02934/) (2020)
- [Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphology](https://mlsanity.com/paper/2112.14656/) (2021)
- [Face.evoLVe: A High-Performance Face Recognition Library](https://mlsanity.com/paper/2107.08621/) (2021)
- [QMagFace: Simple and Accurate Quality-Aware Face Recognition](https://mlsanity.com/paper/2111.13475/) (2021)
- [Face Recognition Accuracy Across Demographics: Shining a Light Into the Problem](https://mlsanity.com/paper/2206.01881/) (2022)
- [Is Face Recognition Sexist? No, Gendered Hairstyles and Biology Are](https://mlsanity.com/paper/2008.06989/) (2020)
- [Rethinking Common Assumptions to Mitigate Racial Bias in Face Recognition Datasets](https://mlsanity.com/paper/2109.03229/) (2021)
- [Does Face Recognition Accuracy Get Better With Age? Deep Face Matchers Say No](https://mlsanity.com/paper/1911.06396/) (2019)
- [The Gender Gap in Face Recognition Accuracy Is a Hairy Problem](https://mlsanity.com/paper/2206.04867/) (2022)
- [Logical Consistency and Greater Descriptive Power for Facial Hair Attribute Learning](https://mlsanity.com/paper/2302.11102/) (2023)

## Built on by (most cited first, 13 of 13)

- [Vec2Face: Scaling Face Dataset Generation with Loosely Constrained Vectors](https://mlsanity.com/paper/2409.02979/) (2024)
- [Review of Demographic Fairness in Face Recognition](https://mlsanity.com/paper/2502.02309/) (2025)
- [Can the accuracy bias by facial hairstyle be reduced through balancing the training data?](https://mlsanity.com/paper/2405.20062/) (2024)
- [Hybrid Generative Fusion for Efficient and Privacy-Preserving Face Recognition Dataset Generation](https://mlsanity.com/paper/2508.10672/) (2025)
- [LogicNet: A Logical Consistency Embedded Face Attribute Learning Network](https://mlsanity.com/paper/2311.11208/) (2023)
- [Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement](https://mlsanity.com/paper/2405.09403/) (2024)
- [Lights, Camera, Matching: The Role of Image Illumination in Fair Face Recognition](https://mlsanity.com/paper/2501.08910/) (2025)
- [On the Illusion of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes](https://mlsanity.com/paper/2501.12020/) (2025)
- [What's color got to do with it? Face recognition in grayscale](https://mlsanity.com/paper/2309.05180/) (2023)
- [Vec2Face+ for Face Dataset Generation](https://mlsanity.com/paper/2507.17192/) (2025)
- [Balancing Beyond Discrete Categories: Continuous Demographic Labels for Fair Face Recognition](https://mlsanity.com/paper/2506.01532/) (2025)
- [Measures of classification bias derived from sample size analysis](https://mlsanity.com/paper/2601.03453/) (2026)
- [On the Use of Synthetic Data for Threshold Calibration in Face Recognition: Performance and Security Implications for Border Control Systems](https://mlsanity.com/paper/2607.25990/) (2026)

## Abstract (arXiv)

The issue of demographic disparities in face recognition accuracy has attracted increasing attention in recent years. Various face image datasets have been proposed as 'fair' or 'balanced' to assess the accuracy of face recognition algorithms across demographics. These datasets typically balance the number of identities and images across demographics. It is important to note that the number of identities and images in an evaluation dataset are em not driving factors for 1-to-1 face matching accuracy. Moreover, balancing the number of identities and images does not ensure balance in other factors known to impact accuracy, such as head pose, brightness, and image quality. We demonstrate these issues using several recently proposed datasets. To improve the ability to perform less biased evaluations, we propose a bias-aware toolkit that facilitates creation of cross-demographic evaluation datasets balanced on factors mentioned in this paper.