# End-to-end 3D face reconstruction with deep neural networks

arXiv: https://arxiv.org/abs/1704.05020 · HTML: https://mlsanity.com/paper/1704.05020/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2017.

## Built on by (most cited first, 30 of 47)

- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset](https://mlsanity.com/paper/2203.14057/) (2022)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement](https://mlsanity.com/paper/2106.11423/) (2021)
- [FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction](https://mlsanity.com/paper/2111.01082/) (2021)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [Towards Practical Capture of High-Fidelity Relightable Avatars](https://mlsanity.com/paper/2309.04247/) (2023)
- [Synthetic Data in Human Analysis: A Survey](https://mlsanity.com/paper/2208.09191/) (2022)
- [FNeVR: Neural Volume Rendering for Face Animation](https://mlsanity.com/paper/2209.10340/) (2022)
- [VRMM: A Volumetric Relightable Morphable Head Model](https://mlsanity.com/paper/2402.04101/) (2024)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization](https://mlsanity.com/paper/2305.03043/) (2023)
- [Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation](https://mlsanity.com/paper/2303.01311/) (2023)
- [TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction](https://mlsanity.com/paper/2502.10982/) (2025)
- [SIDER: Single-Image Neural Optimization for Facial Geometric Detail Recovery](https://mlsanity.com/paper/2108.05465/) (2021)
- [LOTR: Face Landmark Localization Using Localization Transformer](https://mlsanity.com/paper/2109.10057/) (2021)
- [Unsupervised High-Fidelity Facial Texture Generation and Reconstruction](https://mlsanity.com/paper/2110.04760/) (2021)
- [RAFaRe: Learning Robust and Accurate Non-parametric 3D Face Reconstruction from Pseudo 2D&3D Pairs](https://mlsanity.com/paper/2302.05486/) (2023)
- [A Perceptual Shape Loss for Monocular 3D Face Reconstruction](https://mlsanity.com/paper/2310.19580/) (2023)
- [Bring Your Own Character: A Holistic Solution for Automatic Facial Animation Generation of Customized Characters](https://mlsanity.com/paper/2402.13724/) (2024)
- [Generalizable Face Landmarking Guided by Conditional Face Warping](https://mlsanity.com/paper/2404.12322/) (2024)
- [GRAPE: Generalizable and Robust Multi-view Facial Capture](https://mlsanity.com/paper/2407.10193/) (2024)
- [SPARK: Self-supervised Personalized Real-time Monocular Face Capture](https://mlsanity.com/paper/2409.07984/) (2024)
- [Weakly-Supervised Multi-Face 3D Reconstruction](https://mlsanity.com/paper/2101.02000/) (2021)
- [Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape](https://mlsanity.com/paper/2204.04379/) (2022)
- [Implicit Neural Deformation for Sparse-View Face Reconstruction](https://mlsanity.com/paper/2112.02494/) (2021)

## Abstract (arXiv)

Monocular 3D facial shape reconstruction from a single 2D facial image has been an active research area due to its wide applications. Inspired by the success of deep neural networks (DNN), we propose a DNN-based approach for End-to-End 3D FAce Reconstruction (UH-E2FAR) from a single 2D image. Different from recent works that reconstruct and refine the 3D face in an iterative manner using both an RGB image and an initial 3D facial shape rendering, our DNN model is end-to-end, and thus the complicated 3D rendering process can be avoided. Moreover, we integrate in the DNN architecture two components, namely a multi-task loss function and a fusion convolutional neural network (CNN) to improve facial expression reconstruction. With the multi-task loss function, 3D face reconstruction is divided into neutral 3D facial shape reconstruction and expressive 3D facial shape reconstruction. The neutral 3D facial shape is class-specific. Therefore, higher layer features are useful. In comparison, the expressive 3D facial shape favors lower or intermediate layer features. With the fusion-CNN, features from different intermediate layers are fused and transformed for predicting the 3D expressive facial shape. Through extensive experiments, we demonstrate the superiority of our end-to-end framework in improving the accuracy of 3D face reconstruction.