# Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape

arXiv: https://arxiv.org/abs/2204.04379 · HTML: https://mlsanity.com/paper/2204.04379/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 71 references that are in the map.
- First posted on arXiv 9 Apr 2022.

## Builds on (most cited first, 30 of 34)

- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression](https://mlsanity.com/paper/1703.07834/) (2017)
- [Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation](https://mlsanity.com/paper/1703.10131/) (2017)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)
- [End-to-end 3D face reconstruction with deep neural networks](https://mlsanity.com/paper/1704.05020/) (2017)
- [3D Guided Fine-Grained Face Manipulation](https://mlsanity.com/paper/1902.08900/) (2019)
- [Photo-Realistic Facial Details Synthesis from Single Image](https://mlsanity.com/paper/1903.10873/) (2019)
- [Effective Face Frontalization in Unconstrained Images](https://mlsanity.com/paper/1411.7964/) (2014)
- [3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning](https://mlsanity.com/paper/2007.13666/) (2020)
- [3D Face Reconstruction with Geometry Details from a Single Image](https://mlsanity.com/paper/1702.05619/) (2017)
- [Dense Face Alignment](https://mlsanity.com/paper/1709.01442/) (2017)
- [Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders](https://mlsanity.com/paper/1904.03525/) (2019)
- [Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition](https://mlsanity.com/paper/1708.02734/) (2017)
- [Self-supervised Learning of Detailed 3D Face Reconstruction](https://mlsanity.com/paper/1910.11791/) (2019)

## Built on by (most cited first, 5 of 5)

- [3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation](https://mlsanity.com/paper/2312.00311/) (2023)
- [FLNeRF: 3D Facial Landmarks Estimation in Neural Radiance Fields](https://mlsanity.com/paper/2211.11202/) (2022)
- [Recent Advancement in 3D Biometrics using Monocular Camera](https://mlsanity.com/paper/2401.02646/) (2024)
- [TimeWalker: Personalized Neural Space for Lifelong Head Avatars](https://mlsanity.com/paper/2412.02421/) (2024)
- [FaceSnap: Real-Time Personalized Lightstage Facial Performance Capture](https://mlsanity.com/paper/2608.31033/) (2026)

## Abstract (arXiv)

3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. However, previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry, which is attributed to insufficient ground-truth 3D shapes, unreliable training strategies and limited representation power of 3DMM. To alleviate this issue, this paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person. Specifically, given a 2D image as the input, we virtually render the image in several calibrated views to normalize pose variations while preserving the original image geometry. A many-to-one hourglass network serves as the encode-decoder to fuse multiview features and generate vertex displacements as the fine-grained geometry. Besides, the neural network is trained by directly optimizing the visual effect, where two 3D shapes are compared by measuring the similarity between the multiview images rendered from the shapes. Finally, we propose to generate the ground-truth 3D shapes by registering RGB-D images followed by pose and shape augmentation, providing sufficient data for network training. Experiments on several challenging protocols demonstrate the superior reconstruction accuracy of our proposal on the face shape.