# Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics

arXiv: https://arxiv.org/abs/2106.11240 · HTML: https://mlsanity.com/paper/2106.11240/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 45 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 6 of 6)

- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Predictive Inequity in Object Detection](https://mlsanity.com/paper/1902.11097/) (2019)
- [Characterizing the Variability in Face Recognition Accuracy Relative to Race](https://mlsanity.com/paper/1904.07325/) (2019)
- [An Experimental Evaluation of Covariates Effects on Unconstrained Face Verification](https://mlsanity.com/paper/1808.05508/) (2018)
- [Understanding Unequal Gender Classification Accuracy from Face Images](https://mlsanity.com/paper/1812.00099/) (2018)
- [Analysis of Manual and Automated Skin Tone Assignments for Face Recognition Applications](https://mlsanity.com/paper/2104.14685/) (2021)

## Built on by (most cited first, 15 of 15)

- [ITI-GEN: Inclusive Text-to-Image Generation](https://mlsanity.com/paper/2309.05569/) (2023)
- [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023)
- [AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark](https://mlsanity.com/paper/2406.00783/) (2024)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [Face Recognition Accuracy Across Demographics: Shining a Light Into the Problem](https://mlsanity.com/paper/2206.01881/) (2022)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Measuring Hidden Bias within Face Recognition via Racial Phenotypes](https://mlsanity.com/paper/2110.09839/) (2021)
- [Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets](https://mlsanity.com/paper/2305.09072/) (2023)
- [Racial Bias within Face Recognition: A Survey](https://mlsanity.com/paper/2305.00817/) (2023)
- [Exploring Causes of Demographic Variations In Face Recognition Accuracy](https://mlsanity.com/paper/2304.07175/) (2023)
- [Bridging the Data Gap in AI Reliability Research and Establishing DR-AIR, a Comprehensive Data Repository for AI Reliability](https://mlsanity.com/paper/2502.12386/) (2025)
- [Human-Machine Comparison for Cross-Race Face Verification: Race Bias at the Upper Limits of Performance?](https://mlsanity.com/paper/2305.16443/) (2023)
- [Eye Sclera for Fair Face Image Quality Assessment](https://mlsanity.com/paper/2501.07158/) (2025)
- [Demographic Variability in Face Image Quality Measures](https://mlsanity.com/paper/2501.07898/) (2025)
- [Automated Disentangling Analysis of Skin Colour for Lesion Images](https://mlsanity.com/paper/2602.19055/) (2026)

## Abstract (arXiv)

With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups. Recently, phenotypes such as skin-tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin-tone in evaluations of biometric performance or in AI more broadly. In this study, we explore the relationship between face-area-lightness-measures (FALMs) estimated from images and ground-truth skin readings collected using a device designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALM. This variation was only reduced by greater control of acquisition (camera, background, and environment). Next, we compare ground-truth FALM to Fitzpatrick Skin Types (FST) categories obtained using the standard, in-person, medical survey and show FST is poorly predictive of skin-tone. Finally, we show how noisy estimation of FALM leads to errors selecting explanatory factors for demographic differentials. These results demonstrate that measures of skin-tone for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin-tone.