# Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors

arXiv: https://arxiv.org/abs/1807.00966 · HTML: https://mlsanity.com/paper/1807.00966/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2018.

## Built on by (most cited first, 30 of 49)

- [Improving the Efficiency and Robustness of Deepfakes Detection through Precise Geometric Features](https://mlsanity.com/paper/2104.04480/) (2021)
- [STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection](https://mlsanity.com/paper/2306.02763/) (2023)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [Supervision by Registration and Triangulation for Landmark Detection](https://mlsanity.com/paper/2101.09866/) (2021)
- [Sparse to Dense Motion Transfer for Face Image Animation](https://mlsanity.com/paper/2109.00471/) (2021)
- [ZoomNAS: Searching for Whole-body Human Pose Estimation in the Wild](https://mlsanity.com/paper/2208.11547/) (2022)
- [GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation](https://mlsanity.com/paper/2112.01036/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Mask-FPAN: Semi-Supervised Face Parsing in the Wild With De-Occlusion and UV GAN](https://mlsanity.com/paper/2212.09098/) (2022)
- [Precise Facial Landmark Detection by Reference Heatmap Transformer](https://mlsanity.com/paper/2303.07840/) (2023)
- [Semi-supervised Keypoint Localization](https://mlsanity.com/paper/2101.07988/) (2021)
- [Subpixel Heatmap Regression for Facial Landmark Localization](https://mlsanity.com/paper/2111.02360/) (2021)
- [ALEBk: Feasibility Study of Attention Level Estimation via Blink Detection applied to e-Learning](https://mlsanity.com/paper/2112.09165/) (2021)
- [Semi-supervised Anatomical Landmark Detection via Shape-regulated Self-training](https://mlsanity.com/paper/2105.13593/) (2021)
- [UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing](https://mlsanity.com/paper/2108.05650/) (2021)
- [ASMNet: a Lightweight Deep Neural Network for Face Alignment and Pose Estimation](https://mlsanity.com/paper/2103.00119/) (2021)
- [mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection](https://mlsanity.com/paper/2309.07880/) (2023)
- [Deep Two-Stream Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/2110.11680/) (2021)
- [Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning](https://mlsanity.com/paper/2105.05838/) (2021)
- [Facial Landmark Points Detection Using Knowledge Distillation-Based Neural Networks](https://mlsanity.com/paper/2111.07047/) (2021)
- [Robust and Precise Facial Landmark Detection by Self-Calibrated Pose Attention Network](https://mlsanity.com/paper/2112.12328/) (2021)
- [Generalizable Face Landmarking Guided by Conditional Face Warping](https://mlsanity.com/paper/2404.12322/) (2024)
- [Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelines](https://mlsanity.com/paper/2204.10746/) (2022)
- [Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking](https://mlsanity.com/paper/2102.06690/) (2021)
- [landmarker: a Toolkit for Anatomical Landmark Localization in 2D/3D Images](https://mlsanity.com/paper/2501.10098/) (2025)
- [Reliable uncertainty quantification for 2D/3D anatomical landmark localization using multi-output conformal prediction](https://mlsanity.com/paper/2503.14106/) (2025)
- [Reasoning Structural Relation for Occlusion-Robust Facial Landmark Localization](https://mlsanity.com/paper/2112.10087/) (2021)
- [Multi-Domain Multi-Definition Landmark Localization for Small Datasets](https://mlsanity.com/paper/2203.10358/) (2022)
- [Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification](https://mlsanity.com/paper/2312.07338/) (2023)
- [ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection](https://mlsanity.com/paper/2412.13174/) (2024)

## Abstract (arXiv)

In this paper, we present supervision-by-registration, an unsupervised approach to improve the precision of facial landmark detectors on both images and video. Our key observation is that the detections of the same landmark in adjacent frames should be coherent with registration, i.e., optical flow. Interestingly, the coherency of optical flow is a source of supervision that does not require manual labeling, and can be leveraged during detector training. For example, we can enforce in the training loss function that a detected landmark at framet-1 followed by optical flow tracking from framet-1 to framet should coincide with the location of the detection at framet. Essentially, supervision-by-registration augments the training loss function with a registration loss, thus training the detector to have output that is not only close to the annotations in labeled images, but also consistent with registration on large amounts of unlabeled videos. End-to-end training with the registration loss is made possible by a differentiable Lucas-Kanade operation, which computes optical flow registration in the forward pass, and back-propagates gradients that encourage temporal coherency in the detector. The output of our method is a more precise image-based facial landmark detector, which can be applied to single images or video. With supervision-by-registration, we demonstrate (1) improvements in facial landmark detection on both images (300W, ALFW) and video (300VW, Youtube-Celebrities), and (2) significant reduction of jittering in video detections.