# Flexible Portrait Image Editing with Fine-Grained Control

arXiv: https://arxiv.org/abs/2204.01318 · HTML: https://mlsanity.com/paper/2204.01318/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection](https://mlsanity.com/paper/1909.01955/) (2019)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Deep Exemplar-based Colorization](https://mlsanity.com/paper/1807.06587/) (2018)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histograms](https://mlsanity.com/paper/2011.11731/) (2020)
- [Mask-Guided Portrait Editing with Conditional GANs](https://mlsanity.com/paper/1905.10346/) (2019)
- [Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformation](https://mlsanity.com/paper/1812.09912/) (2018)
- [TextureGAN: Controlling Deep Image Synthesis with Texture Patches](https://mlsanity.com/paper/1706.02823/) (2017)
- [MVF-Net: Multi-View 3D Face Morphable Model Regression](https://mlsanity.com/paper/1904.04473/) (2019)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)
- [Towards High Fidelity Face Relighting with Realistic Shadows](https://mlsanity.com/paper/2104.00825/) (2021)

## Built on by (most cited first, 1 of 1)

- [Semantics-Preserving Sketch Embedding for Face Generation](https://mlsanity.com/paper/2211.13015/) (2022)

## Abstract (arXiv)

We develop a new method for portrait image editing, which supports fine-grained editing of geometries, colors, lights and shadows using a single neural network model. We adopt a novel asymmetric conditional GAN architecture: the generators take the transformed conditional inputs, such as edge maps, color palette, sliders and masks, that can be directly edited by the user; the discriminators take the conditional inputs in the way that can guide controllable image generation more effectively. Taking color editing as an example, we feed color palettes (which can be edited easily) into the generator, and color maps (which contain positional information of colors) into the discriminator. We also design a region-weighted discriminator so that higher weights are assigned to more important regions, like eyes and skin. Using a color palette, the user can directly specify the desired colors of hair, skin, eyes, lip and background. Color sliders allow the user to blend colors in an intuitive manner. The user can also edit lights and shadows by modifying the corresponding masks. We demonstrate the effectiveness of our method by evaluating it on the CelebAMask-HQ dataset with a wide range of tasks, including geometry/color/shadow/light editing, hand-drawn sketch to image translation, and color transfer. We also present ablation studies to justify our design.