# HIH: Towards More Accurate Face Alignment via Heatmap in Heatmap

arXiv: https://arxiv.org/abs/2104.03100 · HTML: https://mlsanity.com/paper/2104.03100/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 82 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [RetinaFace: Single-stage Dense Face Localisation in the Wild](https://mlsanity.com/paper/1905.00641/) (2019)
- [Distribution-Aware Coordinate Representation for Human Pose Estimation](https://mlsanity.com/paper/1910.06278/) (2019)
- [Bottom-up Object Detection by Grouping Extreme and Center Points](https://mlsanity.com/paper/1901.08043/) (2019)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Towards Accurate Multi-person Pose Estimation in the Wild](https://mlsanity.com/paper/1701.01779/) (2017)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood](https://mlsanity.com/paper/2004.02980/) (2020)
- [Pixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the Wild](https://mlsanity.com/paper/2003.03771/) (2020)
- [Style Aggregated Network for Facial Landmark Detection](https://mlsanity.com/paper/1803.04108/) (2018)
- [Structured Landmark Detection via Topology-Adapting Deep Graph Learning](https://mlsanity.com/paper/2004.08190/) (2020)
- [DeCaFA: Deep Convolutional Cascade for Face Alignment In The Wild](https://mlsanity.com/paper/1904.02549/) (2019)
- [Deep Alignment Network: A convolutional neural network for robust face alignment](https://mlsanity.com/paper/1706.01789/) (2017)

## Built on by (most cited first, 15 of 15)

- [STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection](https://mlsanity.com/paper/2306.02763/) (2023)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models](https://mlsanity.com/paper/2304.00600/) (2023)
- [LOTR: Face Landmark Localization Using Localization Transformer](https://mlsanity.com/paper/2109.10057/) (2021)
- [Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection](https://mlsanity.com/paper/2411.07167/) (2024)
- [OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis](https://mlsanity.com/paper/2506.02891/) (2025)
- [Automated Detection of Cat Facial Landmarks](https://mlsanity.com/paper/2310.09793/) (2023)
- [Self-supervised Auxiliary Learning for Texture and Model-based Hybrid Robust and Fair Featuring in Face Analysis](https://mlsanity.com/paper/2409.19582/) (2024)
- [Multi-Domain Multi-Definition Landmark Localization for Small Datasets](https://mlsanity.com/paper/2203.10358/) (2022)
- [Fiducial Focus Augmentation for Facial Landmark Detection](https://mlsanity.com/paper/2402.15044/) (2024)
- [ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection](https://mlsanity.com/paper/2412.13174/) (2024)
- [A survey and classification of face alignment methods based on face models](https://mlsanity.com/paper/2311.03082/) (2023)
- [OccFace: Unified Occlusion-Aware Facial Landmark Detection with Per-Point Visibility](https://mlsanity.com/paper/2602.10728/) (2026)
- [A Backbone Benchmarking Study on Self-supervised Learning as a Auxiliary Task with Texture-based Local Descriptors for Face Analysis](https://mlsanity.com/paper/2603.22190/) (2026)
- [WorldComp2D: Spatio-semantic Representations of Object Identity and Location from Local Views](https://mlsanity.com/paper/2605.11743/) (2026)

## Abstract (arXiv)

Heatmap-based regression overcomes the lack of spatial and contextual information of direct coordinate regression, and has revolutionized the task of face alignment. Yet it suffers from quantization errors caused by neglecting subpixel coordinates in image resizing and network downsampling. In this paper, we first quantitatively analyze the quantization error on benchmarks, which accounts for more than 1/3 of the whole prediction errors for state-of-the-art methods. To tackle this problem, we propose a novel Heatmap In Heatmap(HIH) representation and a coordinate soft-classification (CSC) method, which are seamlessly integrated into the classic hourglass network. The HIH representation utilizes nested heatmaps to jointly represent the coordinate label: one heatmap called integer heatmap stands for the integer coordinate, and the other heatmap named decimal heatmap represents the subpixel coordinate. The range of a decimal heatmap makes up one pixel in the corresponding integer heatmap. Besides, we transfer the offset regression problem to an interval classification task, and CSC regards the confidence of the pixel as the probability of the interval. Meanwhile, CSC applying the distribution loss leverage the soft labels generated from the Gaussian distribution function to guide the offset heatmap training, which makes it easier to learn the distribution of coordinate offsets. Extensive experiments on challenging benchmark datasets demonstrate that our HIH can achieve state-of-the-art results. In particular, our HIH reaches 4.08 NME (Normalized Mean Error) on WFLW, and 3.21 on COFW, which exceeds previous methods by a significant margin.