# Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection

arXiv: https://arxiv.org/abs/2405.20117 · HTML: https://mlsanity.com/paper/2405.20117/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 21 of 21)

- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [Facial Landmark Detection: a Literature Survey](https://mlsanity.com/paper/1805.05563/) (2018)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding](https://mlsanity.com/paper/2206.03484/) (2022)
- [Faster Than Real-time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Poses](https://mlsanity.com/paper/1707.05653/) (2017)
- [Fast Facial Landmark Detection and Applications: A Survey](https://mlsanity.com/paper/2101.10808/) (2021)
- [Facial Feature Point Detection: A Comprehensive Survey](https://mlsanity.com/paper/1410.1037/) (2014)
- [3D Morphable Models as Spatial Transformer Networks](https://mlsanity.com/paper/1708.07199/) (2017)
- [3D Dense Face Alignment with Fused Features by Aggregating CNNs and GCNs](https://mlsanity.com/paper/2203.04643/) (2022)
- [A lightweight 3D dense facial landmark estimation model from position map data](https://mlsanity.com/paper/2308.15170/) (2023)

## Built on by (most cited first, 2 of 2)

- [GNM Head: A Generative aNthropometric Model of the human head](https://mlsanity.com/paper/2607.23687/) (2026)
- [Joint Learning of Depth and Appearance for Portrait Image Animation](https://mlsanity.com/paper/2501.08649/) (2025)

## Abstract (arXiv)

In this paper, we examine 3 important issues in the practical use of state-of-the-art facial landmark detectors and show how a combination of specific architectural modifications can directly improve their accuracy and temporal stability. First, many facial landmark detectors require face normalization as a preprocessing step, which is accomplished by a separately-trained neural network that crops and resizes the face in the input image. There is no guarantee that this pre-trained network performs the optimal face normalization for landmark detection. We instead analyze the use of a spatial transformer network that is trained alongside the landmark detector in an unsupervised manner, and jointly learn optimal face normalization and landmark detection. Second, we show that modifying the output head of the landmark predictor to infer landmarks in a canonical 3D space can further improve accuracy. To convert the predicted 3D landmarks into screen-space, we additionally predict the camera intrinsics and head pose from the input image. As a side benefit, this allows to predict the 3D face shape from a given image only using 2D landmarks as supervision, which is useful in determining landmark visibility among other things. Finally, when training a landmark detector on multiple datasets at the same time, annotation inconsistencies across datasets forces the network to produce a suboptimal average. We propose to add a semantic correction network to address this issue. This additional lightweight neural network is trained alongside the landmark detector, without requiring any additional supervision. While the insights of this paper can be applied to most common landmark detectors, we specifically target a recently-proposed continuous 2D landmark detector to demonstrate how each of our additions leads to meaningful improvements over the state-of-the-art on standard benchmarks.