# Analysis of Manual and Automated Skin Tone Assignments for Face Recognition Applications

arXiv: https://arxiv.org/abs/2104.14685 · HTML: https://mlsanity.com/paper/2104.14685/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021)
- [Diversity in Faces](https://mlsanity.com/paper/1901.10436/) (2019)
- [Characterizing the Variability in Face Recognition Accuracy Relative to Race](https://mlsanity.com/paper/1904.07325/) (2019)
- [An Experimental Evaluation of Covariates Effects on Unconstrained Face Verification](https://mlsanity.com/paper/1808.05508/) (2018)
- [Understanding Unequal Gender Classification Accuracy from Face Images](https://mlsanity.com/paper/1812.00099/) (2018)
- [Jointly De-biasing Face Recognition and Demographic Attribute Estimation](https://mlsanity.com/paper/1911.08080/) (2019)
- [SREDS: A dichromatic separation based measure of skin color](https://mlsanity.com/paper/2104.02926/) (2021)

## Built on by (most cited first, 16 of 16)

- [Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm](https://mlsanity.com/paper/2207.02942/) (2022)
- [AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark](https://mlsanity.com/paper/2406.00783/) (2024)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Measuring Hidden Bias within Face Recognition via Racial Phenotypes](https://mlsanity.com/paper/2110.09839/) (2021)
- [Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics](https://mlsanity.com/paper/2106.11240/) (2021)
- [Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets](https://mlsanity.com/paper/2305.09072/) (2023)
- [Racial Bias within Face Recognition: A Survey](https://mlsanity.com/paper/2305.00817/) (2023)
- [TrueSkin: Towards Fair and Accurate Skin Tone Recognition and Generation](https://mlsanity.com/paper/2509.10980/) (2025)
- [Exploring Causes of Demographic Variations In Face Recognition Accuracy](https://mlsanity.com/paper/2304.07175/) (2023)
- [What's color got to do with it? Face recognition in grayscale](https://mlsanity.com/paper/2309.05180/) (2023)
- [Evaluating Fairness and Mitigating Bias in Machine Learning: A Novel Technique using Tensor Data and Bayesian Regression](https://mlsanity.com/paper/2506.11627/) (2025)
- [Multi-Focus Image Fusion Based on Spatial Frequency(SF) and Consistency Verification(CV) in DCT Domain](https://mlsanity.com/paper/2305.11265/) (2023)
- [Peepers & Pixels: Human Recognition Accuracy on Low Resolution Faces](https://mlsanity.com/paper/2503.20108/) (2025)
- [Mitigating Individual Skin Tone Bias in Skin Lesion Classification through Distribution-Aware Reweighting](https://mlsanity.com/paper/2512.08733/) (2025)
- [Skin Tone in the Wild: A Dataset and Benchmark for 10-tone MST Classification](https://mlsanity.com/paper/2603.02475/) (2026)
- [Fuzzy Accuracy Compensates for Label Subjectivity in Classification of Skin Tone Using Wearable Photoplethysmography Signals](https://mlsanity.com/paper/2608.18969/) (2026)

## Abstract (arXiv)

News reports have suggested that darker skin tone causes an increase in face recognition errors. The Fitzpatrick scale is widely used in dermatology to classify sensitivity to sun exposure and skin tone. In this paper, we analyze a set of manual Fitzpatrick skin type assignments and also employ the individual typology angle to automatically estimate the skin tone from face images. The set of manual skin tone rating experiments shows that there are inconsistencies between human raters that are difficult to eliminate. Efforts to automate skin tone rating suggest that it is particularly challenging on images collected without a calibration object in the scene. However, after the color-correction, the level of agreement between automated and manual approaches is found to be 96% or better for the MORPH images. To our knowledge, this is the first work to: (a) examine the consistency of manual skin tone ratings across observers, (b) document that there is substantial variation in the rating of the same image by different observers even when exemplar images are given for guidance and all images are color-corrected, and (c) compare manual versus automated skin tone ratings.