# Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation

arXiv: https://arxiv.org/abs/2205.03962 · HTML: https://mlsanity.com/paper/2205.03962/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 77 references that are in the map.
- First posted on arXiv 8 May 2022.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Demographic Bias in Biometrics: A Survey on an Emerging Challenge](https://mlsanity.com/paper/2003.02488/) (2020)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Face Recognition: Too Bias, or Not Too Bias?](https://mlsanity.com/paper/2002.06483/) (2020)
- [FML: Face Model Learning from Videos](https://mlsanity.com/paper/1812.07603/) (2018)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [Diversity in Faces](https://mlsanity.com/paper/1901.10436/) (2019)
- [Predictive Inequity in Object Detection](https://mlsanity.com/paper/1902.11097/) (2019)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Disentangling Factors of Variation Using Few Labels](https://mlsanity.com/paper/1905.01258/) (2019)
- [A Morphable Face Albedo Model](https://mlsanity.com/paper/2004.02711/) (2020)
- [Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks](https://mlsanity.com/paper/2003.05653/) (2020)
- [Photo-Realistic Facial Details Synthesis from Single Image](https://mlsanity.com/paper/1903.10873/) (2019)

## Built on by (most cited first, 30 of 32)

- [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://mlsanity.com/paper/2404.01030/) (2024)
- [ITI-GEN: Inclusive Text-to-Image Generation](https://mlsanity.com/paper/2309.05569/) (2023)
- [FLARE: Fast Learning of Animatable and Relightable Mesh Avatars](https://mlsanity.com/paper/2310.17519/) (2023)
- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [Relightify: Relightable 3D Faces from a Single Image via Diffusion Models](https://mlsanity.com/paper/2305.06077/) (2023)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Beyond Aesthetics: Cultural Competence in Text-to-Image Models](https://mlsanity.com/paper/2407.06863/) (2024)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Beyond Skin Tone: A Multidimensional Measure of Apparent Skin Color](https://mlsanity.com/paper/2309.05148/) (2023)
- [ReCap: Better Gaussian Relighting with Cross-Environment Captures](https://mlsanity.com/paper/2412.07534/) (2024)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [3D Face Reconstruction: the Road to Forensics](https://mlsanity.com/paper/2309.11357/) (2023)
- [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024)
- [FairQueue: Rethinking Prompt Learning for Fair Text-to-Image Generation](https://mlsanity.com/paper/2410.18615/) (2024)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction](https://mlsanity.com/paper/2409.09740/) (2024)
- [A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization](https://mlsanity.com/paper/2310.03205/) (2023)
- [FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models](https://mlsanity.com/paper/2312.04465/) (2023)
- [Monocular Identity-Conditioned Facial Reflectance Reconstruction](https://mlsanity.com/paper/2404.00301/) (2024)
- [Bias Amplification in Stable Diffusion's Representation of Stigma Through Skin Tones and Their Homogeneity](https://mlsanity.com/paper/2508.17465/) (2025)
- [SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images](https://mlsanity.com/paper/2312.06085/) (2023)
- [Latent Directions: A Simple Pathway to Bias Mitigation in Generative AI](https://mlsanity.com/paper/2406.06352/) (2024)
- [High-Fidelity Facial Albedo Estimation via Texture Quantization](https://mlsanity.com/paper/2406.13149/) (2024)
- [Remote Bio-Sensing: Open Source Benchmark Framework for Fair Evaluation of rPPG](https://mlsanity.com/paper/2307.12644/) (2023)
- [Democratizing the Creation of Animatable Facial Avatars](https://mlsanity.com/paper/2401.16534/) (2024)
- [Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators](https://mlsanity.com/paper/2411.12002/) (2024)
- [WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering](https://mlsanity.com/paper/2512.11237/) (2025)
- [FiCA: Feed-forward instant Gaussian Codec Avatars from a Single Portrait Image](https://mlsanity.com/paper/2606.24232/) (2026)
- [Reproducibility Study of "ITI-GEN: Inclusive Text-to-Image Generation"](https://mlsanity.com/paper/2407.19996/) (2024)

## Abstract (arXiv)

Virtual facial avatars will play an increasingly important role in immersive communication, games and the metaverse, and it is therefore critical that they be inclusive. This requires accurate recovery of the appearance, represented by albedo, regardless of age, sex, or ethnicity. While significant progress has been made on estimating 3D facial geometry, albedo estimation has received less attention. The task is fundamentally ambiguous because the observed color is a function of albedo and lighting, both of which are unknown. We find that current methods are biased towards light skin tones due to (1) strongly biased priors that prefer lighter pigmentation and (2) algorithmic solutions that disregard the light/albedo ambiguity. To address this, we propose a new evaluation dataset (FAIR) and an algorithm (TRUST) to improve albedo estimation and, hence, fairness. Specifically, we create the first facial albedo evaluation benchmark where subjects are balanced in terms of skin color, and measure accuracy using the Individual Typology Angle (ITA) metric. We then address the light/albedo ambiguity by building on a key observation: the image of the full scene -- as opposed to a cropped image of the face -- contains important information about lighting that can be used for disambiguation. TRUST regresses facial albedo by conditioning both on the face region and a global illumination signal obtained from the scene image. Our experimental results show significant improvement compared to state-of-the-art methods on albedo estimation, both in terms of accuracy and fairness. The evaluation benchmark and code will be made available for research purposes at https://trust.is.tue.mpg.de.