# Generating Animatable 3D Cartoon Faces from Single Portraits

arXiv: https://arxiv.org/abs/2307.01468 · HTML: https://mlsanity.com/paper/2307.01468/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 4 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains](https://mlsanity.com/paper/2010.05334/) (2020)
- [Effective Face Frontalization in Unconstrained Images](https://mlsanity.com/paper/1411.7964/) (2014)
- [Fitting a 3D Morphable Model to Edges: A Comparison Between Hard and Soft Correspondences](https://mlsanity.com/paper/1602.01125/) (2016)
- [3DCaricShop: A Dataset and A Baseline Method for Single-view 3D Caricature Face Reconstruction](https://mlsanity.com/paper/2103.08204/) (2021)
- [Alive Caricature from 2D to 3D](https://mlsanity.com/paper/1803.06802/) (2018)
- [Landmark Detection and 3D Face Reconstruction for Caricature using a Nonlinear Parametric Model](https://mlsanity.com/paper/2004.09190/) (2020)
- [Self-supervised Low Light Image Enhancement and Denoising](https://mlsanity.com/paper/2103.00832/) (2021)
- [InverseFaceNet: Deep Monocular Inverse Face Rendering](https://mlsanity.com/paper/1703.10956/) (2017)

## Abstract (arXiv)

With the booming of virtual reality (VR) technology, there is a growing need for customized 3D avatars. However, traditional methods for 3D avatar modeling are either time-consuming or fail to retain similarity to the person being modeled. We present a novel framework to generate animatable 3D cartoon faces from a single portrait image. We first transfer an input real-world portrait to a stylized cartoon image with a StyleGAN. Then we propose a two-stage reconstruction method to recover the 3D cartoon face with detailed texture, which first makes a coarse estimation based on template models, and then refines the model by non-rigid deformation under landmark supervision. Finally, we propose a semantic preserving face rigging method based on manually created templates and deformation transfer. Compared with prior arts, qualitative and quantitative results show that our method achieves better accuracy, aesthetics, and similarity criteria. Furthermore, we demonstrate the capability of real-time facial animation of our 3D model.