# ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection

arXiv: https://arxiv.org/abs/2412.13174 · HTML: https://mlsanity.com/paper/2412.13174/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Towards Robust Blind Face Restoration with Codebook Lookup Transformer](https://mlsanity.com/paper/2206.11253/) (2022)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood](https://mlsanity.com/paper/2004.02980/) (2020)
- [Pixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the Wild](https://mlsanity.com/paper/2003.03771/) (2020)
- [Numerical Coordinate Regression with Convolutional Neural Networks](https://mlsanity.com/paper/1801.07372/) (2018)
- [6D Rotation Representation For Unconstrained Head Pose Estimation](https://mlsanity.com/paper/2202.12555/) (2022)
- [Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors](https://mlsanity.com/paper/1807.00966/) (2018)
- [Sparse Local Patch Transformer for Robust Face Alignment and Landmarks Inherent Relation Learning](https://mlsanity.com/paper/2203.06541/) (2022)
- [Going Deeper in Facial Expression Recognition using Deep Neural Networks](https://mlsanity.com/paper/1511.04110/) (2015)
- [STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection](https://mlsanity.com/paper/2306.02763/) (2023)
- [ADNet: Leveraging Error-Bias Towards Normal Direction in Face Alignment](https://mlsanity.com/paper/2109.05721/) (2021)
- [Towards Accurate Facial Landmark Detection via Cascaded Transformers](https://mlsanity.com/paper/2208.10808/) (2022)
- [Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models](https://mlsanity.com/paper/2304.00600/) (2023)
- [PFLD: A Practical Facial Landmark Detector](https://mlsanity.com/paper/1902.10859/) (2019)
- [Multi-task head pose estimation in-the-wild](https://mlsanity.com/paper/2202.02299/) (2022)
- [HIH: Towards More Accurate Face Alignment via Heatmap in Heatmap](https://mlsanity.com/paper/2104.03100/) (2021)
- [Subpixel Heatmap Regression for Facial Landmark Localization](https://mlsanity.com/paper/2111.02360/) (2021)
- [RePFormer: Refinement Pyramid Transformer for Robust Facial Landmark Detection](https://mlsanity.com/paper/2207.03917/) (2022)

## Built on by (most cited first, 2 of 2)

- [OccFace: Unified Occlusion-Aware Facial Landmark Detection with Per-Point Visibility](https://mlsanity.com/paper/2602.10728/) (2026)
- [DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding](https://mlsanity.com/paper/2610.17015/) (2026)

## Abstract (arXiv)

Although facial landmark detection (FLD) has gained significant progress, existing FLD methods still suffer from performance drops on partially non-visible faces, such as faces with occlusions or under extreme lighting conditions or poses. To address this issue, we introduce ORFormer, a novel transformer-based method that can detect non-visible regions and recover their missing features from visible parts. Specifically, ORFormer associates each image patch token with one additional learnable token called the messenger token. The messenger token aggregates features from all but its patch. This way, the consensus between a patch and other patches can be assessed by referring to the similarity between its regular and messenger embeddings, enabling non-visible region identification. Our method then recovers occluded patches with features aggregated by the messenger tokens. Leveraging the recovered features, ORFormer compiles high-quality heatmaps for the downstream FLD task. Extensive experiments show that our method generates heatmaps resilient to partial occlusions. By integrating the resultant heatmaps into existing FLD methods, our method performs favorably against the state of the arts on challenging datasets such as WFLW and COFW.