# 3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation

arXiv: https://arxiv.org/abs/2312.00311 · HTML: https://mlsanity.com/paper/2312.00311/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 65 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration](https://mlsanity.com/paper/1605.03344/) (2016)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion](https://mlsanity.com/paper/2111.12702/) (2021)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research](https://mlsanity.com/paper/1911.05063/) (2019)
- [FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset](https://mlsanity.com/paper/2203.14057/) (2022)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://mlsanity.com/paper/1907.06724/) (2019)
- [A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images](https://mlsanity.com/paper/2302.14434/) (2023)
- [Learning Complete 3D Morphable Face Models from Images and Videos](https://mlsanity.com/paper/2010.01679/) (2020)
- [Face Parsing with RoI Tanh-Warping](https://mlsanity.com/paper/1906.01342/) (2019)
- [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://mlsanity.com/paper/2203.09729/) (2022)
- [DAD-3DHeads: A Large-scale Dense, Accurate and Diverse Dataset for 3D Head Alignment from a Single Image](https://mlsanity.com/paper/2204.03688/) (2022)
- [Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing](https://mlsanity.com/paper/2203.14448/) (2022)

## Built on by (most cited first, 30 of 38)

- [TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction](https://mlsanity.com/paper/2502.10982/) (2025)
- [RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Images](https://mlsanity.com/paper/2409.03644/) (2024)
- [DynamicFace: High-Quality and Consistent Face Swapping for Image and Video using Composable 3D Facial Priors](https://mlsanity.com/paper/2501.08553/) (2025)
- [DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning](https://mlsanity.com/paper/2504.14509/) (2025)
- [DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer](https://mlsanity.com/paper/2601.01425/) (2026)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation](https://mlsanity.com/paper/2503.14295/) (2025)
- [S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch](https://mlsanity.com/paper/2408.01218/) (2024)
- [SEREP: Semantic Facial Expression Representation for Robust In-the-Wild Capture and Retargeting](https://mlsanity.com/paper/2412.14371/) (2024)
- [STAvatar: Soft Binding and Temporal Density Control for Monocular 3D Head Avatars Reconstruction](https://mlsanity.com/paper/2511.19854/) (2025)
- [VGGTFace: Topologically Consistent Facial Geometry Reconstruction in the Wild](https://mlsanity.com/paper/2511.20366/) (2025)
- [SRM-Hair: Single Image Head Mesh Reconstruction via 3D Morphable Hair](https://mlsanity.com/paper/2503.06154/) (2025)
- [High-Quality Facial Albedo Generation for 3D Face Reconstruction from a Single Image using a Coarse-to-Fine Approach](https://mlsanity.com/paper/2506.13233/) (2025)
- [Driver-Net: Multi-Camera Fusion for Assessing Driver Take-Over Readiness in Automated Vehicles](https://mlsanity.com/paper/2507.04139/) (2025)
- [BFSM: 3D Bidirectional Face-Skull Morphable Model](https://mlsanity.com/paper/2509.24577/) (2025)
- [AvatarTex: High-Fidelity Facial Texture Reconstruction from Single-Image Stylized Avatars](https://mlsanity.com/paper/2511.06721/) (2025)
- [MANGO:Natural Multi-speaker 3D Talking Head Generation via 2D-Lifted Enhancement](https://mlsanity.com/paper/2601.01749/) (2026)
- [Skullptor: High Fidelity 3D Head Reconstruction in Seconds with Multi-View Normal Prediction](https://mlsanity.com/paper/2602.21100/) (2026)
- [Any3DAvatar: Fast and High-Quality Full-Head 3D Avatar Reconstruction from Single Portrait Image](https://mlsanity.com/paper/2604.13856/) (2026)
- [Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading](https://mlsanity.com/paper/2606.28144/) (2026)
- [High-Quality 3D Head Reconstruction from Any Single Portrait Image](https://mlsanity.com/paper/2503.08516/) (2025)
- [Improving Facial Rig Semantics for Tracking and Retargeting](https://mlsanity.com/paper/2508.08429/) (2025)
- [Hierarchical MLANet: Multi-level Attention for 3D Face Reconstruction From Single Images](https://mlsanity.com/paper/2509.10024/) (2025)
- [MSMA: Multi-Scale Feature Fusion For Multi-Attribute 3D Face Reconstruction From Unconstrained Images](https://mlsanity.com/paper/2509.11763/) (2025)
- [GLVD: Guided Learned Vertex Descent](https://mlsanity.com/paper/2510.06046/) (2025)
- [Exploring Specular Reflection Inconsistency for Generalizable Face Forgery Detection](https://mlsanity.com/paper/2602.06452/) (2026)
- [Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features](https://mlsanity.com/paper/2603.20012/) (2026)
- [Profile-Specific 3DMM Regression from a Single Lateral Face Image](https://mlsanity.com/paper/2605.01746/) (2026)
- [FaceParts: Segmentation and Editing of Gaussian Splatting](https://mlsanity.com/paper/2605.13853/) (2026)
- [OMGTex: One-stage Multi-style Facial Texture Reconstruction without Geometry Guidance](https://mlsanity.com/paper/2605.25778/) (2026)

## Abstract (arXiv)

3D Morphable Models (3DMMs) provide promising 3D face reconstructions in various applications. However, existing methods struggle to reconstruct faces with extreme expressions due to deficiencies in supervisory signals, such as sparse or inaccurate landmarks. Segmentation information contains effective geometric contexts for face reconstruction. Certain attempts intuitively depend on differentiable renderers to compare the rendered silhouettes of reconstruction with segmentation, which is prone to issues like local optima and gradient instability. In this paper, we fully utilize the facial part segmentation geometry by introducing Part Re-projection Distance Loss (PRDL). Specifically, PRDL transforms facial part segmentation into 2D points and re-projects the reconstruction onto the image plane. Subsequently, by introducing grid anchors and computing different statistical distances from these anchors to the point sets, PRDL establishes geometry descriptors to optimize the distribution of the point sets for face reconstruction. PRDL exhibits a clear gradient compared to the renderer-based methods and presents state-of-the-art reconstruction performance in extensive quantitative and qualitative experiments. Our project is available at https://github.com/wang-zidu/3DDFA-V3 .