mlsanityData to 7 Oct 2026

Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

arXiv 2302.13279 ↗First posted 26 Feb 2023cs.CV cs.GRcited by 8

Not on the map. Why?

Abstract (arXiv)

Facial makeup enriches the beauty of not only real humans but also virtual characters; therefore, makeup for 3D facial models is highly in demand in productions. However, painting directly on 3D faces and capturing real-world makeup are costly, and extracting makeup from 2D images often struggles with shading effects and occlusions. This paper presents the first method for extracting makeup for 3D facial models from a single makeup portrait. Our method consists of the following three steps. First, we exploit the strong prior of 3D morphable models via regression-based inverse rendering to extract coarse materials such as geometry and diffuse/specular albedos that are represented in the UV space. Second, we refine the coarse materials, which may have missing pixels due to occlusions. We apply inpainting and optimization. Finally, we extract the bare skin, makeup, and an alpha matte from the diffuse albedo. Our method offers various applications for not only 3D facial models but also 2D portrait images. The extracted makeup is well-aligned in the UV space, from which we build a large-scale makeup dataset and a parametric makeup model for 3D faces. Our disentangled materials also yield robust makeup transfer and illumination-aware makeup interpolation/removal without a reference image.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 48 of its 73 references that are in the map. First posted on arXiv 26 Feb 2023.

Builds on

48 of 73 references.

References · 48

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  8. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  9. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  10. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  11. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  12. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
36 more
  1. 2021GAN Inversion: A Surveycited by 366
  2. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  3. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  4. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  5. 2022Towards Metrical Reconstruction of Human Facescited by 165
  6. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  7. 2019Single Image Portrait Relightingcited by 152
  8. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  9. 2017Morphable Face Models - An Open Frameworkcited by 138
  10. 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
  11. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  12. 2018Unsupervised Training for 3D Morphable Model Regressioncited by 99
  13. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  14. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  15. 2020Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistencycited by 78
  16. 2022Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptationcited by 67
  17. 2019PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfercited by 66
  18. 2019LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeupcited by 63
  19. 2017UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognitioncited by 62
  20. 2020A Morphable Face Albedo Modelcited by 54
  21. 2020High-Fidelity 3D Digital Human Head Creation from RGB-D Selfiescited by 47
  22. 2020OSTeC: One-Shot Texture Completioncited by 37
  23. 2022VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relightingcited by 37
  24. 2021Practical Face Reconstruction via Differentiable Ray Tracingcited by 35
  25. 2021Lipstick ain't enough: Beyond Color Matching for In-the-Wild Makeup Transfercited by 35
  26. 2022Geometry-aware Single-image Full-body Human Relightingcited by 34
  27. 2016Photorealistic Facial Texture Inference Using Deep Neural Networkscited by 32
  28. 2022De-rendering 3D Objects in the Wildcited by 31
  29. 2021Single-image Full-body Human Relightingcited by 30
  30. 2022EleGANt: Exquisite and Locally Editable GAN for Makeup Transfercited by 28
  31. 2021SOGAN: 3D-Aware Shadow and Occlusion Robust GAN for Makeup Transfercited by 22
  32. 2021Relighting Humans in the Wild: Monocular Full-Body Human Relighting with Domain Adaptationcited by 21
  33. 2019Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body Human Imagescited by 16
  34. 2020CA-GAN: Weakly Supervised Color Aware GAN for Controllable Makeup Transfercited by 15
  35. 2022BareSkinNet: De-makeup and De-lighting via 3D Face Reconstructioncited by 10
  36. 2022Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Rendererscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 3; 2026: 1.

8 papers.

  1. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  2. 2024Makeup Prior Models for 3D Facial Makeup Estimation and Applicationscited by 7
  3. 2025FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategycited by 7
  4. 2024BeautyBank: Encoding Facial Makeup in Latent Spacecited by 4
  5. 2025AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatarscited by 3
  6. 2025FFHQ-Makeup: Paired Synthetic Makeup Dataset with Facial Consistency Across Multiple Stylescited by 2
  7. 2026Supervised makeup transfer with a curated dataset: Decoupling identity and makeup features for enhanced transformationcited by 1
  8. 2023MicroGlam: Microscopic Skin Image Dataset with Cosmeticscited by 0