# Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

arXiv: https://arxiv.org/abs/2110.09772 · HTML: https://mlsanity.com/paper/2110.09772/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 78 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [Rethinking Pseudo-LiDAR Representation](https://mlsanity.com/paper/2008.04582/) (2020)
- [Grid-GCN for Fast and Scalable Point Cloud Learning](https://mlsanity.com/paper/1912.02984/) (2019)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [FML: Face Model Learning from Videos](https://mlsanity.com/paper/1812.07603/) (2018)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression](https://mlsanity.com/paper/1703.07834/) (2017)
- [Style Aggregated Network for Facial Landmark Detection](https://mlsanity.com/paper/1803.04108/) (2018)
- [MVF-Net: Multi-View 3D Face Morphable Model Regression](https://mlsanity.com/paper/1904.04473/) (2019)

## Built on by (most cited first, 27 of 27)

- [DAD-3DHeads: A Large-scale Dense, Accurate and Diverse Dataset for 3D Head Alignment from a Single Image](https://mlsanity.com/paper/2204.03688/) (2022)
- [HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details](https://mlsanity.com/paper/2303.11225/) (2023)
- [Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues](https://mlsanity.com/paper/2402.00281/) (2024)
- [ImFace++: A Sophisticated Nonlinear 3D Morphable Face Model with Implicit Neural Representations](https://mlsanity.com/paper/2312.04028/) (2023)
- [DEGAS: Detailed Expressions on Full-Body Gaussian Avatars](https://mlsanity.com/paper/2408.10588/) (2024)
- [Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?](https://mlsanity.com/paper/2203.09824/) (2022)
- [DirectMHP: Direct 2D Multi-Person Head Pose Estimation with Full-range Angles](https://mlsanity.com/paper/2302.01110/) (2023)
- [DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignment](https://mlsanity.com/paper/2305.11522/) (2023)
- [A Perceptual Shape Loss for Monocular 3D Face Reconstruction](https://mlsanity.com/paper/2310.19580/) (2023)
- [EFHQ: Multi-purpose ExtremePose-Face-HQ dataset](https://mlsanity.com/paper/2312.17205/) (2023)
- [VGGHeads: 3D Multi Head Alignment with a Large-Scale Synthetic Dataset](https://mlsanity.com/paper/2407.18245/) (2024)
- [Joint Multi-Person Body Detection and Orientation Estimation via One Unified Embedding](https://mlsanity.com/paper/2210.15586/) (2022)
- [Meta-Optimization for Higher Model Generalizability in Single-Image Depth Prediction](https://mlsanity.com/paper/2305.07269/) (2023)
- [FLNeRF: 3D Facial Landmarks Estimation in Neural Radiance Fields](https://mlsanity.com/paper/2211.11202/) (2022)
- [Semi-Supervised Unconstrained Head Pose Estimation in the Wild](https://mlsanity.com/paper/2404.02544/) (2024)
- [FaceLift: Semi-supervised 3D Facial Landmark Localization](https://mlsanity.com/paper/2405.19646/) (2024)
- [A Real-Time Multi-Task Learning System for Joint Detection of Face, Facial Landmark and Head Pose](https://mlsanity.com/paper/2309.11773/) (2023)
- [InSpaceType: Reconsider Space Type in Indoor Monocular Depth Estimation](https://mlsanity.com/paper/2309.13516/) (2023)
- [A survey and classification of face alignment methods based on face models](https://mlsanity.com/paper/2311.03082/) (2023)
- [3D Face Reconstruction Error Decomposed: A Modular Benchmark for Fair and Fast Method Evaluation](https://mlsanity.com/paper/2505.18025/) (2025)
- [3D Face Alignment Through Fusion of Head Pose Information and Features](https://mlsanity.com/paper/2308.13327/) (2023)
- [Student Classroom Behavior Detection based on Spatio-Temporal Network and Multi-Model Fusion](https://mlsanity.com/paper/2310.16267/) (2023)
- [On the power of data augmentation for head pose estimation](https://mlsanity.com/paper/2407.05357/) (2024)
- [Bidirectional Regression for Monocular 6DoF Head Pose Estimation and Reference System Alignment](https://mlsanity.com/paper/2407.14136/) (2024)
- [Geometry-Aware Video Inpainting for Joint Headset Occlusion Removal and Face Reconstruction in Social XR](https://mlsanity.com/paper/2508.12336/) (2025)
- [Profile-Specific 3DMM Regression from a Single Lateral Face Image](https://mlsanity.com/paper/2605.01746/) (2026)
- [FADRA: Frequency-Aware Diffusion with Residual Adaptation for Video Face Restoration](https://mlsanity.com/paper/2607.06389/) (2026)

## Abstract (arXiv)

This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks to predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling. Our synergy process leverages a representation cycle for 3DMM parameters and 3D landmarks. 3D landmarks can be extracted and refined from face meshes built by 3DMM parameters. We next reverse the representation direction and show that predicting 3DMM parameters from sparse 3D landmarks improves the information flow. Together we create a synergy process that utilizes the relation between 3D landmarks and 3DMM parameters, and they collaboratively contribute to better performance. We extensively validate our contribution on full tasks of facial geometry prediction and show our superior and robust performance on these tasks for various scenarios. Particularly, we adopt only simple and widely-used network operations to attain fast and accurate facial geometry prediction. Codes and data: https://choyingw.github.io/works/SynergyNet/