# Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition arXiv: https://arxiv.org/abs/2302.13279 ยท HTML: https://mlsanity.com/paper/2302.13279/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 73 references that are in the map. - First posted on arXiv 26 Feb 2023. ## Builds on (most cited first, 30 of 48) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019) - [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017) - [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016) - [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017) - [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018) - [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017) - [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019) - [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020) - [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021) - [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019) - [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022) - [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019) - [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022) - [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017) - [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019) - [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019) - [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017) - [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017) - [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016) - [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018) - [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017) - [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020) - [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020) - [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022) - [PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer](https://mlsanity.com/paper/1909.06956/) (2019) - [LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup](https://mlsanity.com/paper/1904.11272/) (2019) ## Built on by (most cited first, 8 of 8) - [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023) - [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024) - [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025) - [BeautyBank: Encoding Facial Makeup in Latent Space](https://mlsanity.com/paper/2411.11231/) (2024) - [AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars](https://mlsanity.com/paper/2507.02419/) (2025) - [FFHQ-Makeup: Paired Synthetic Makeup Dataset with Facial Consistency Across Multiple Styles](https://mlsanity.com/paper/2508.03241/) (2025) - [Supervised makeup transfer with a curated dataset: Decoupling identity and makeup features for enhanced transformation](https://mlsanity.com/paper/2602.00729/) (2026) - [MicroGlam: Microscopic Skin Image Dataset with Cosmetics](https://mlsanity.com/paper/2401.05339/) (2023) ## Abstract (arXiv) Facial makeup enriches the beauty of not only real humans but also virtual characters; therefore, makeup for 3D facial models is highly in demand in productions. However, painting directly on 3D faces and capturing real-world makeup are costly, and extracting makeup from 2D images often struggles with shading effects and occlusions. This paper presents the first method for extracting makeup for 3D facial models from a single makeup portrait. Our method consists of the following three steps. First, we exploit the strong prior of 3D morphable models via regression-based inverse rendering to extract coarse materials such as geometry and diffuse/specular albedos that are represented in the UV space. Second, we refine the coarse materials, which may have missing pixels due to occlusions. We apply inpainting and optimization. Finally, we extract the bare skin, makeup, and an alpha matte from the diffuse albedo. Our method offers various applications for not only 3D facial models but also 2D portrait images. The extracted makeup is well-aligned in the UV space, from which we build a large-scale makeup dataset and a parametric makeup model for 3D faces. Our disentangled materials also yield robust makeup transfer and illumination-aware makeup interpolation/removal without a reference image.