# Enhancing the Authenticity of Rendered Portraits with Identity-Consistent Transfer Learning

arXiv: https://arxiv.org/abs/2310.04194 · HTML: https://mlsanity.com/paper/2310.04194/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 69 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 30 of 36)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [Image2StyleGAN++: How to Edit the Embedded Images?](https://mlsanity.com/paper/1911.11544/) (2019)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement](https://mlsanity.com/paper/2104.02699/) (2021)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Few-shot Image Generation via Cross-domain Correspondence](https://mlsanity.com/paper/2104.06820/) (2021)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video](https://mlsanity.com/paper/2210.06108/) (2022)
- [Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains](https://mlsanity.com/paper/2010.05334/) (2020)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [MoFaNeRF: Morphable Facial Neural Radiance Field](https://mlsanity.com/paper/2112.02308/) (2021)
- [GIRAFFE HD: A High-Resolution 3D-aware Generative Model](https://mlsanity.com/paper/2203.14954/) (2022)
- [Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks](https://mlsanity.com/paper/2110.08398/) (2021)
- [Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer](https://mlsanity.com/paper/2203.13248/) (2022)

## Built on by (most cited first, 1 of 1)

- [FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models](https://mlsanity.com/paper/2410.14429/) (2024)

## Abstract (arXiv)

Despite rapid advances in computer graphics, creating high-quality photo-realistic virtual portraits is prohibitively expensive. Furthermore, the well-know ''uncanny valley'' effect in rendered portraits has a significant impact on the user experience, especially when the depiction closely resembles a human likeness, where any minor artifacts can evoke feelings of eeriness and repulsiveness. In this paper, we present a novel photo-realistic portrait generation framework that can effectively mitigate the ''uncanny valley'' effect and improve the overall authenticity of rendered portraits. Our key idea is to employ transfer learning to learn an identity-consistent mapping from the latent space of rendered portraits to that of real portraits. During the inference stage, the input portrait of an avatar can be directly transferred to a realistic portrait by changing its appearance style while maintaining the facial identity. To this end, we collect a new dataset, Daz-Rendered-Faces-HQ (DRFHQ), that is specifically designed for rendering-style portraits. We leverage this dataset to fine-tune the StyleGAN2 generator, using our carefully crafted framework, which helps to preserve the geometric and color features relevant to facial identity. We evaluate our framework using portraits with diverse gender, age, and race variations. Qualitative and quantitative evaluations and ablation studies show the advantages of our method compared to state-of-the-art approaches.