# TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone

arXiv: https://arxiv.org/abs/2505.20637 · HTML: https://mlsanity.com/paper/2505.20637/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 23 of 23)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Demographic Bias in Biometrics: A Survey on an Emerging Challenge](https://mlsanity.com/paper/2003.02488/) (2020)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021)
- [Diversity in Faces](https://mlsanity.com/paper/1901.10436/) (2019)
- [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023)
- [Meta Balanced Network for Fair Face Recognition](https://mlsanity.com/paper/2205.06548/) (2022)
- [One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision](https://mlsanity.com/paper/2102.02320/) (2021)
- [Estimating Skin Tone and Effects on Classification Performance in Dermatology Datasets](https://mlsanity.com/paper/1910.13268/) (2019)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Beyond Skin Tone: A Multidimensional Measure of Apparent Skin Color](https://mlsanity.com/paper/2309.05148/) (2023)
- [Human Skin Detection Using RGB, HSV and YCbCr Color Models](https://mlsanity.com/paper/1708.02694/) (2017)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [Bias and Fairness on Multimodal Emotion Detection Algorithms](https://mlsanity.com/paper/2205.08383/) (2022)
- [Achieve Fairness without Demographics for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2401.08066/) (2024)
- [Racial Bias within Face Recognition: A Survey](https://mlsanity.com/paper/2305.00817/) (2023)
- [The Impact of Racial Distribution in Training Data on Face Recognition Bias: A Closer Look](https://mlsanity.com/paper/2211.14498/) (2022)
- [Focused LRP: Explainable AI for Face Morphing Attack Detection](https://mlsanity.com/paper/2103.14697/) (2021)
- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [Addressing Racial Bias in Facial Emotion Recognition](https://mlsanity.com/paper/2308.04674/) (2023)
- [Towards Fairness in AI for Melanoma Detection: Systemic Review and Recommendations](https://mlsanity.com/paper/2411.12846/) (2024)

## Abstract (arXiv)

Understanding how facial affect analysis (FAA) systems perform across different demographic groups requires reliable measurement of sensitive attributes such as ancestry, often approximated by skin tone, which itself is highly influenced by lighting conditions. This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness (L^*) and Hue (H^*). Using AffectNet and a MobileNet-based model, we assess fairness across skin tone groups defined by each method. Results reveal a severe underrepresentation of dark skin tones (~ 2 %), alongside fairness disparities in F1-score (up to 0.08) and TPR (up to 0.11) across groups. While ITA shows limitations due to its sensitivity to lighting, the H^*-L^* method yields more consistent subgrouping and enables clearer diagnostics through metrics such as Equal Opportunity. Grad-CAM analysis further highlights differences in model attention patterns by skin tone, suggesting variation in feature encoding. To support future mitigation efforts, we also propose a modular fairness-aware pipeline that integrates perceptual skin tone estimation, model interpretability, and fairness evaluation. These findings emphasize the relevance of skin tone measurement choices in fairness assessment and suggest that ITA-based evaluations may overlook disparities affecting darker-skinned individuals.