# An Efficient Multitask Neural Network for Face Alignment, Head Pose Estimation and Face Tracking

arXiv: https://arxiv.org/abs/2103.07615 · HTML: https://mlsanity.com/paper/2103.07615/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 13 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition](https://mlsanity.com/paper/1603.01249/) (2016)
- [Style Aggregated Network for Facial Landmark Detection](https://mlsanity.com/paper/1803.04108/) (2018)
- [End-to-end 3D face reconstruction with deep neural networks](https://mlsanity.com/paper/1704.05020/) (2017)
- [DeCaFA: Deep Convolutional Cascade for Face Alignment In The Wild](https://mlsanity.com/paper/1904.02549/) (2019)
- [Effective Face Frontalization in Unconstrained Images](https://mlsanity.com/paper/1411.7964/) (2014)
- [Aggregation via Separation: Boosting Facial Landmark Detector with Semi-Supervised Style Translation](https://mlsanity.com/paper/1908.06440/) (2019)
- [3FabRec: Fast Few-shot Face alignment by Reconstruction](https://mlsanity.com/paper/1911.10448/) (2019)
- [Supervision by Registration and Triangulation for Landmark Detection](https://mlsanity.com/paper/2101.09866/) (2021)
- [Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resources](https://mlsanity.com/paper/1703.00862/) (2017)
- [FAB: A Robust Facial Landmark Detection Framework for Motion-Blurred Videos](https://mlsanity.com/paper/1910.12100/) (2019)
- [PropagationNet: Propagate Points to Curve to Learn Structure Information](https://mlsanity.com/paper/2006.14308/) (2020)
- [Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regressions](https://mlsanity.com/paper/1603.09732/) (2016)

## Built on by (most cited first, 2 of 2)

- [MC-ViViT: Multi-branch Classifier-ViViT to detect Mild Cognitive Impairment in older adults using facial videos](https://mlsanity.com/paper/2304.05292/) (2023)
- [PedRecNet: Multi-task deep neural network for full 3D human pose and orientation estimation](https://mlsanity.com/paper/2204.11548/) (2022)

## Abstract (arXiv)

While Convolutional Neural Networks (CNNs) have significantly boosted the performance of face related algorithms, maintaining accuracy and efficiency simultaneously in practical use remains challenging. The state-of-the-art methods employ deeper networks for better performance, which makes it less practical for mobile applications because of more parameters and higher computational complexity. Therefore, we propose an efficient multitask neural network, Alignment & Tracking & Pose Network (ATPN) for face alignment, face tracking and head pose estimation. Specifically, to achieve better performance with fewer layers for face alignment, we introduce a shortcut connection between shallow-layer and deep-layer features. We find the shallow-layer features are highly correspond to facial boundaries that can provide the structural information of face and it is crucial for face alignment. Moreover, we generate a cheap heatmap based on the face alignment result and fuse it with features to improve the performance of the other two tasks. Based on the heatmap, the network can utilize both geometric information of landmarks and appearance information for head pose estimation. The heatmap also provides attention clues for face tracking. The face tracking task also saves us the face detection procedure for each frame, which also significantly boost the real-time capability for video-based tasks. We experimentally validate ATPN on four benchmark datasets, WFLW, 300VW, WIDER Face and 300W-LP. The experimental results demonstrate that it achieves better performance with much less parameters and lower computational complexity compared to other light models.