# Lipstick ain't enough: Beyond Color Matching for In-the-Wild Makeup Transfer

arXiv: https://arxiv.org/abs/2104.01867 · HTML: https://mlsanity.com/paper/2104.01867/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer](https://mlsanity.com/paper/1909.06956/) (2019)
- [3D Face Reconstruction by Learning from Synthetic Data](https://mlsanity.com/paper/1609.04387/) (2016)
- [LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup](https://mlsanity.com/paper/1904.11272/) (2019)
- [Photo-Realistic Facial Details Synthesis from Single Image](https://mlsanity.com/paper/1903.10873/) (2019)
- [CA-GAN: Weakly Supervised Color Aware GAN for Controllable Makeup Transfer](https://mlsanity.com/paper/2008.10298/) (2020)
- [Disentangled Makeup Transfer with Generative Adversarial Network](https://mlsanity.com/paper/1907.01144/) (2019)

## Built on by (most cited first, 30 of 35)

- [Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Model](https://mlsanity.com/paper/2403.07764/) (2024)
- [SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal](https://mlsanity.com/paper/2112.03631/) (2021)
- [BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfer](https://mlsanity.com/paper/2212.05855/) (2022)
- [Content-Style Decoupling for Unsupervised Makeup Transfer without Generating Pseudo Ground Truth](https://mlsanity.com/paper/2405.17240/) (2024)
- [GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Image](https://mlsanity.com/paper/2404.02152/) (2024)
- [BareSkinNet: De-makeup and De-lighting via 3D Face Reconstruction](https://mlsanity.com/paper/2209.09029/) (2022)
- [SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models](https://mlsanity.com/paper/2412.11058/) (2024)
- [Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition](https://mlsanity.com/paper/2302.13279/) (2023)
- [Makeup-Guided Facial Privacy Protection via Untrained Neural Network Priors](https://mlsanity.com/paper/2408.12387/) (2024)
- [Deep Face Video Inpainting via UV Mapping](https://mlsanity.com/paper/2109.00681/) (2021)
- [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024)
- [MAD: Makeup All-in-One with Cross-Domain Diffusion Model](https://mlsanity.com/paper/2504.02545/) (2025)
- [FLUX-Makeup: High-Fidelity, Identity-Consistent, and Robust Makeup Transfer via Diffusion Transformer](https://mlsanity.com/paper/2508.05069/) (2025)
- [BeautyBank: Encoding Facial Makeup in Latent Space](https://mlsanity.com/paper/2411.11231/) (2024)
- [Modern Augmented Reality: Applications, Trends, and Future Directions](https://mlsanity.com/paper/2202.09450/) (2022)
- [A comprehensive survey on semantic facial attribute editing using generative adversarial networks](https://mlsanity.com/paper/2205.10587/) (2022)
- [Let's Get the FACS Straight -- Reconstructing Obstructed Facial Features](https://mlsanity.com/paper/2311.05221/) (2023)
- [AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars](https://mlsanity.com/paper/2507.02419/) (2025)
- [EvoMakeup: High-Fidelity and Controllable Makeup Editing with MakeupQuad](https://mlsanity.com/paper/2508.05994/) (2025)
- [Robust Deepfake On Unrestricted Media: Generation And Detection](https://mlsanity.com/paper/2202.06228/) (2022)
- [Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers](https://mlsanity.com/paper/2205.06305/) (2022)
- [SARA: Controllable Makeup Transfer with Spatial Alignment and Region-Adaptive Normalization](https://mlsanity.com/paper/2311.16828/) (2023)
- [FFHQ-Makeup: Paired Synthetic Makeup Dataset with Facial Consistency Across Multiple Styles](https://mlsanity.com/paper/2508.03241/) (2025)
- [Flexible Portrait Image Editing with Fine-Grained Control](https://mlsanity.com/paper/2204.01318/) (2022)
- [Gorgeous: Create Your Desired Character Facial Makeup from Any Ideas](https://mlsanity.com/paper/2404.13944/) (2024)
- [V-LASIK: Consistent Glasses-Removal from Videos Using Synthetic Data](https://mlsanity.com/paper/2406.14510/) (2024)
- [Supervised makeup transfer with a curated dataset: Decoupling identity and makeup features for enhanced transformation](https://mlsanity.com/paper/2602.00729/) (2026)
- [Enhancing Face Recognition with Latent Space Data Augmentation and Facial Posture Reconstruction](https://mlsanity.com/paper/2301.11986/) (2023)
- [Scalable and Realistic Virtual Try-on Application for Foundation Makeup with Kubelka-Munk Theory](https://mlsanity.com/paper/2507.07333/) (2025)
- [DiffClean: Diffusion-based Makeup Removal for Accurate Age Estimation](https://mlsanity.com/paper/2507.13292/) (2025)

## Abstract (arXiv)

Makeup transfer is the task of applying on a source face the makeup style from a reference image. Real-life makeups are diverse and wild, which cover not only color-changing but also patterns, such as stickers, blushes, and jewelries. However, existing works overlooked the latter components and confined makeup transfer to color manipulation, focusing only on light makeup styles. In this work, we propose a holistic makeup transfer framework that can handle all the mentioned makeup components. It consists of an improved color transfer branch and a novel pattern transfer branch to learn all makeup properties, including color, shape, texture, and location. To train and evaluate such a system, we also introduce new makeup datasets for real and synthetic extreme makeup. Experimental results show that our framework achieves the state of the art performance on both light and extreme makeup styles. Code is available at https://github.com/VinAIResearch/CPM.