# Faster Than Real-time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Poses

arXiv: https://arxiv.org/abs/1707.05653 · HTML: https://mlsanity.com/paper/1707.05653/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2017.

## Built on by (most cited first, 19 of 19)

- [SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction](https://mlsanity.com/paper/2106.03021/) (2021)
- [Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry](https://mlsanity.com/paper/2110.09772/) (2021)
- [Multi-task head pose estimation in-the-wild](https://mlsanity.com/paper/2202.02299/) (2022)
- [DeepFace-EMD: Re-ranking Using Patch-wise Earth Mover's Distance Improves Out-Of-Distribution Face Identification](https://mlsanity.com/paper/2112.04016/) (2021)
- [Facial Attribute Transformers for Precise and Robust Makeup Transfer](https://mlsanity.com/paper/2104.02894/) (2021)
- [Learning Free-Form Deformation for 3D Face Reconstruction from In-The-Wild Images](https://mlsanity.com/paper/2105.14857/) (2021)
- [DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignment](https://mlsanity.com/paper/2305.11522/) (2023)
- [Weakly-Supervised Multi-Face 3D Reconstruction](https://mlsanity.com/paper/2101.02000/) (2021)
- [Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape](https://mlsanity.com/paper/2204.04379/) (2022)
- [3D Dense Face Alignment with Fused Features by Aggregating CNNs and GCNs](https://mlsanity.com/paper/2203.04643/) (2022)
- [Child Palm-ID: Contactless Palmprint Recognition for Children](https://mlsanity.com/paper/2305.05161/) (2023)
- [FaceLift: Semi-supervised 3D Facial Landmark Localization](https://mlsanity.com/paper/2405.19646/) (2024)
- [Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection](https://mlsanity.com/paper/2405.20117/) (2024)
- [Probabilistic Ranking-Aware Ensembles for Enhanced Object Detections](https://mlsanity.com/paper/2105.03139/) (2021)
- [A Review of 3D Face Reconstruction From a Single Image](https://mlsanity.com/paper/2110.09299/) (2021)
- [A lightweight 3D dense facial landmark estimation model from position map data](https://mlsanity.com/paper/2308.15170/) (2023)
- [A Real-Time Multi-Task Learning System for Joint Detection of Face, Facial Landmark and Head Pose](https://mlsanity.com/paper/2309.11773/) (2023)
- [Accurate 3D Facial Geometry Prediction by Multi-Task, Multi-Modal, and Multi-Representation Landmark Refinement Network](https://mlsanity.com/paper/2104.08403/) (2021)
- [Robust 3D Face Alignment with Multi-Path Neural Architecture Search](https://mlsanity.com/paper/2406.07873/) (2024)

## Abstract (arXiv)

Facial alignment involves finding a set of landmark points on an image with a known semantic meaning. However, this semantic meaning of landmark points is often lost in 2D approaches where landmarks are either moved to visible boundaries or ignored as the pose of the face changes. In order to extract consistent alignment points across large poses, the 3D structure of the face must be considered in the alignment step. However, extracting a 3D structure from a single 2D image usually requires alignment in the first place. We present our novel approach to simultaneously extract the 3D shape of the face and the semantically consistent 2D alignment through a 3D Spatial Transformer Network (3DSTN) to model both the camera projection matrix and the warping parameters of a 3D model. By utilizing a generic 3D model and a Thin Plate Spline (TPS) warping function, we are able to generate subject specific 3D shapes without the need for a large 3D shape basis. In addition, our proposed network can be trained in an end-to-end framework on entirely synthetic data from the 300W-LP dataset. Unlike other 3D methods, our approach only requires one pass through the network resulting in a faster than real-time alignment. Evaluations of our model on the Annotated Facial Landmarks in the Wild (AFLW) and AFLW2000-3D datasets show our method achieves state-of-the-art performance over other 3D approaches to alignment.