# Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition

arXiv: https://arxiv.org/abs/1708.02734 · HTML: https://mlsanity.com/paper/1708.02734/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2017.

## Built on by (most cited first, 21 of 21)

- [Riggable 3D Face Reconstruction via In-Network Optimization](https://mlsanity.com/paper/2104.03493/) (2021)
- [SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction](https://mlsanity.com/paper/2106.03021/) (2021)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [Facial Depth and Normal Estimation using Single Dual-Pixel Camera](https://mlsanity.com/paper/2111.12928/) (2021)
- [2D GANs Meet Unsupervised Single-view 3D Reconstruction](https://mlsanity.com/paper/2207.10183/) (2022)
- [ACR Loss: Adaptive Coordinate-based Regression Loss for Face Alignment](https://mlsanity.com/paper/2203.15835/) (2022)
- [Weakly-Supervised Multi-Face 3D Reconstruction](https://mlsanity.com/paper/2101.02000/) (2021)
- [Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape](https://mlsanity.com/paper/2204.04379/) (2022)
- [FreeEnricher: Enriching Face Landmarks without Additional Cost](https://mlsanity.com/paper/2212.09525/) (2022)
- [INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields](https://mlsanity.com/paper/2312.16197/) (2023)
- [Reasoning Structural Relation for Occlusion-Robust Facial Landmark Localization](https://mlsanity.com/paper/2112.10087/) (2021)
- [Unpaired Multi-domain Attribute Translation of 3D Facial Shapes with a Square and Symmetric Geometric Map](https://mlsanity.com/paper/2308.13245/) (2023)
- [Realistic face animation generation from videos](https://mlsanity.com/paper/2103.14984/) (2021)
- [Towards Fine-grained 3D Face Dense Registration: An Optimal Dividing and Diffusing Method](https://mlsanity.com/paper/2109.11204/) (2021)
- [BabyNet: Reconstructing 3D faces of babies from uncalibrated photographs](https://mlsanity.com/paper/2203.05908/) (2022)
- [Advancing 3D finger knuckle recognition via deep feature learning](https://mlsanity.com/paper/2301.02934/) (2023)
- [3D Face Modeling via Weakly-supervised Disentanglement Network joint Identity-consistency Prior](https://mlsanity.com/paper/2404.16536/) (2024)
- [CSANet: Channel Spatial Attention Network for Robust 3D Face Alignment and Reconstruction](https://mlsanity.com/paper/2405.19659/) (2024)
- [Pixel Sampling for Style Preserving Face Pose Editing](https://mlsanity.com/paper/2106.07310/) (2021)
- [From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion](https://mlsanity.com/paper/2204.03842/) (2022)
- [Improving 2D face recognition via fine-level facial depth generation and RGB-D complementary feature learning](https://mlsanity.com/paper/2305.04426/) (2023)

## Abstract (arXiv)

Face alignment and 3D face reconstruction are traditionally accomplished as separated tasks. By exploring the strong correlation between 2D landmarks and 3D shapes, in contrast, we propose a joint face alignment and 3D face reconstruction method to simultaneously solve these two problems for 2D face images of arbitrary poses and expressions. This method, based on a summation model of 3D faces and cascaded regression in 2D and 3D shape spaces, iteratively and alternately applies two cascaded regressors, one for updating 2D landmarks and the other for 3D shape. The 3D shape and the landmarks are correlated via a 3D-to-2D mapping matrix, which is updated in each iteration to refine the location and visibility of 2D landmarks. Unlike existing methods, the proposed method can fully automatically generate both pose-and-expression-normalized (PEN) and expressive 3D faces and localize both visible and invisible 2D landmarks. Based on the PEN 3D faces, we devise a method to enhance face recognition accuracy across poses and expressions. Both linear and nonlinear implementations of the proposed method are presented and evaluated in this paper. Extensive experiments show that the proposed method can achieve the state-of-the-art accuracy in both face alignment and 3D face reconstruction, and benefit face recognition owing to its reconstructed PEN 3D face.