# Hybrid Approach for 3D Head Reconstruction: Using Neural Networks and Visual Geometry

arXiv: https://arxiv.org/abs/2104.13710 · HTML: https://mlsanity.com/paper/2104.13710/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 28 Apr 2021.

## Builds on (most cited first, 23 of 23)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression](https://mlsanity.com/paper/1703.07834/) (2017)
- [3D Face Reconstruction by Learning from Synthetic Data](https://mlsanity.com/paper/1609.04387/) (2016)
- [Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation](https://mlsanity.com/paper/1703.10131/) (2017)
- [End-to-end 3D face reconstruction with deep neural networks](https://mlsanity.com/paper/1704.05020/) (2017)
- [MVF-Net: Multi-View 3D Face Morphable Model Regression](https://mlsanity.com/paper/1904.04473/) (2019)
- [Marr Revisited: 2D-3D Alignment via Surface Normal Prediction](https://mlsanity.com/paper/1604.01347/) (2016)
- [Training Deep Face Recognition Systems with Synthetic Data](https://mlsanity.com/paper/1802.05891/) (2018)
- [Cross-modal Deep Face Normals with Deactivable Skip Connections](https://mlsanity.com/paper/2003.09691/) (2020)
- [Fitting a 3D Morphable Model to Edges: A Comparison Between Hard and Soft Correspondences](https://mlsanity.com/paper/1602.01125/) (2016)
- [Improving Face Detection Performance with 3D-Rendered Synthetic Data](https://mlsanity.com/paper/1812.07363/) (2018)
- [3D Face Reconstruction from Light Field Images: A Model-free Approach](https://mlsanity.com/paper/1711.05953/) (2017)
- [Digital Twin: Acquiring High-Fidelity 3D Avatar from a Single Image](https://mlsanity.com/paper/1912.03455/) (2019)
- [Landmark Weighting for 3DMM Shape Fitting](https://mlsanity.com/paper/1808.05399/) (2018)

## Abstract (arXiv)

Recovering the 3D geometric structure of a face from a single input image is a challenging active research area in computer vision. In this paper, we present a novel method for reconstructing 3D heads from a single or multiple image(s) using a hybrid approach based on deep learning and geometric techniques. We propose an encoder-decoder network based on the U-net architecture and trained on synthetic data only. It predicts both pixel-wise normal vectors and landmarks maps from a single input photo. Landmarks are used for the pose computation and the initialization of the optimization problem, which, in turn, reconstructs the 3D head geometry by using a parametric morphable model and normal vector fields. State-of-the-art results are achieved through qualitative and quantitative evaluation tests on both single and multi-view settings. Despite the fact that the model was trained only on synthetic data, it successfully recovers 3D geometry and precise poses for real-world images.