# Precise Facial Landmark Detection by Reference Heatmap Transformer

arXiv: https://arxiv.org/abs/2303.07840 · HTML: https://mlsanity.com/paper/2303.07840/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 69 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [TransPose: Keypoint Localization via Transformer](https://mlsanity.com/paper/2012.14214/) (2020)
- [Image Super-Resolution by Neural Texture Transfer](https://mlsanity.com/paper/1903.00834/) (2019)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [Deep Imbalanced Learning for Face Recognition and Attribute Prediction](https://mlsanity.com/paper/1806.00194/) (2018)
- [LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood](https://mlsanity.com/paper/2004.02980/) (2020)
- [Style Aggregated Network for Facial Landmark Detection](https://mlsanity.com/paper/1803.04108/) (2018)
- [Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors](https://mlsanity.com/paper/1807.00966/) (2018)
- [DeCaFA: Deep Convolutional Cascade for Face Alignment In The Wild](https://mlsanity.com/paper/1904.02549/) (2019)
- [Improving Landmark Localization with Semi-Supervised Learning](https://mlsanity.com/paper/1709.01591/) (2017)
- [ADNet: Leveraging Error-Bias Towards Normal Direction in Face Alignment](https://mlsanity.com/paper/2109.05721/) (2021)
- [Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection](https://mlsanity.com/paper/1908.02116/) (2019)
- [Dynamic Attention-controlled Cascaded Shape Regression Exploiting Training Data Augmentation and Fuzzy-set Sample Weighting](https://mlsanity.com/paper/1611.05396/) (2016)
- [Quantized Densely Connected U-Nets for Efficient Landmark Localization](https://mlsanity.com/paper/1808.02194/) (2018)
- [Disentangling 3D Pose in A Dendritic CNN for Unconstrained 2D Face Alignment](https://mlsanity.com/paper/1802.06713/) (2018)
- [Semantic Alignment: Finding Semantically Consistent Ground-truth for Facial Landmark Detection](https://mlsanity.com/paper/1903.10661/) (2019)
- [Laplace Landmark Localization](https://mlsanity.com/paper/1903.11633/) (2019)
- [Robust Face Alignment by Multi-order High-precision Hourglass Network](https://mlsanity.com/paper/2010.08722/) (2020)
- [Robust Facial Landmark Detection by Multi-order Multi-constraint Deep Networks](https://mlsanity.com/paper/2012.04927/) (2020)
- [Subpixel Heatmap Regression for Facial Landmark Localization](https://mlsanity.com/paper/2111.02360/) (2021)

## Built on by (most cited first, 16 of 16)

- [End-to-end Video Gaze Estimation via Capturing Head-face-eye Spatial-temporal Interaction Context](https://mlsanity.com/paper/2310.18131/) (2023)
- [Precise Facial Landmark Detection by Dynamic Semantic Aggregation Transformer](https://mlsanity.com/paper/2412.00740/) (2024)
- [Proto-Former: Unified Facial Landmark Detection by Prototype Transformer](https://mlsanity.com/paper/2510.15338/) (2025)
- [ClusIR: Towards Cluster-Guided All-in-One Image Restoration](https://mlsanity.com/paper/2512.10948/) (2025)
- [KeyPosS: Plug-and-Play Facial Landmark Detection through GPS-Inspired True-Range Multilateration](https://mlsanity.com/paper/2305.16437/) (2023)
- [Rethinking Self-training for Semi-supervised Landmark Detection: A Selection-free Approach](https://mlsanity.com/paper/2404.04556/) (2024)
- [QAFE-Net: Quality Assessment of Facial Expressions with Landmark Heatmaps](https://mlsanity.com/paper/2312.00856/) (2023)
- [POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Search](https://mlsanity.com/paper/2410.09583/) (2024)
- [Identity-Focused Inference and Extraction Attacks on Diffusion Models](https://mlsanity.com/paper/2410.10177/) (2024)
- [Supervision-by-Hallucination-and-Transfer: A Weakly-Supervised Approach for Robust and Precise Facial Landmark Detection](https://mlsanity.com/paper/2601.12919/) (2026)
- [FreqFLD: Towards All-in-One Facial Landmark Detection via Frequency Modulation](https://mlsanity.com/paper/2609.10278/) (2026)
- [Coordinated Transformer with Position & Sample-aware Central Loss for Anatomical Landmark Detection](https://mlsanity.com/paper/2305.11338/) (2023)
- [Adaptive Query Prompting for Multi-Domain Landmark Detection](https://mlsanity.com/paper/2404.01194/) (2024)
- [Dual-Stream Collaborative Transformer for Image Captioning](https://mlsanity.com/paper/2601.12926/) (2026)
- [WorldComp2D: Spatio-semantic Representations of Object Identity and Location from Local Views](https://mlsanity.com/paper/2605.11743/) (2026)
- [FAHCD-Net: Frequency-Adaptive Heatmap-Conditional Diffusion Networks for Robust Facial Landmark Detection](https://mlsanity.com/paper/2609.16842/) (2026)

## Abstract (arXiv)

Most facial landmark detection methods predict landmarks by mapping the input facial appearance features to landmark heatmaps and have achieved promising results. However, when the face image is suffering from large poses, heavy occlusions and complicated illuminations, they cannot learn discriminative feature representations and effective facial shape constraints, nor can they accurately predict the value of each element in the landmark heatmap, limiting their detection accuracy. To address this problem, we propose a novel Reference Heatmap Transformer (RHT) by introducing reference heatmap information for more precise facial landmark detection. The proposed RHT consists of a Soft Transformation Module (STM) and a Hard Transformation Module (HTM), which can cooperate with each other to encourage the accurate transformation of the reference heatmap information and facial shape constraints. Then, a Multi-Scale Feature Fusion Module (MSFFM) is proposed to fuse the transformed heatmap features and the semantic features learned from the original face images to enhance feature representations for producing more accurate target heatmaps. To the best of our knowledge, this is the first study to explore how to enhance facial landmark detection by transforming the reference heatmap information. The experimental results from challenging benchmark datasets demonstrate that our proposed method outperforms the state-of-the-art methods in the literature.