# Facial Landmark Points Detection Using Knowledge Distillation-Based Neural Networks

arXiv: https://arxiv.org/abs/2111.07047 · HTML: https://mlsanity.com/paper/2111.07047/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 72 references that are in the map.
- First posted on arXiv 13 Nov 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [Style Aggregated Network for Facial Landmark Detection](https://mlsanity.com/paper/1803.04108/) (2018)
- [Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors](https://mlsanity.com/paper/1807.00966/) (2018)
- [Deep Alignment Network: A convolutional neural network for robust face alignment](https://mlsanity.com/paper/1706.01789/) (2017)
- [Effective Face Frontalization in Unconstrained Images](https://mlsanity.com/paper/1411.7964/) (2014)
- [Improving Fast Segmentation With Teacher-student Learning](https://mlsanity.com/paper/1810.08476/) (2018)
- [PFLD: A Practical Facial Landmark Detector](https://mlsanity.com/paper/1902.10859/) (2019)
- [Surpassing Human-Level Face Verification Performance on LFW with GaussianFace](https://mlsanity.com/paper/1404.3840/) (2014)
- [Dynamic Attention-controlled Cascaded Shape Regression Exploiting Training Data Augmentation and Fuzzy-set Sample Weighting](https://mlsanity.com/paper/1611.05396/) (2016)
- [Disentangling 3D Pose in A Dendritic CNN for Unconstrained 2D Face Alignment](https://mlsanity.com/paper/1802.06713/) (2018)
- [Interspecies Knowledge Transfer for Facial Keypoint Detection](https://mlsanity.com/paper/1704.04023/) (2017)
- [Simultaneous Facial Landmark Detection, Pose and Deformation Estimation under Facial Occlusion](https://mlsanity.com/paper/1709.08130/) (2017)
- [Joint Multi-view Face Alignment in the Wild](https://mlsanity.com/paper/1708.06023/) (2017)
- [Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation](https://mlsanity.com/paper/1511.07356/) (2015)
- [ASMNet: a Lightweight Deep Neural Network for Face Alignment and Pose Estimation](https://mlsanity.com/paper/2103.00119/) (2021)
- [Face Alignment using a 3D Deeply-initialized Ensemble of Regression Trees](https://mlsanity.com/paper/1902.01831/) (2019)
- [Gaussian Vector: An Efficient Solution for Facial Landmark Detection](https://mlsanity.com/paper/2010.01318/) (2020)
- [An Empirical Study of Recent Face Alignment Methods](https://mlsanity.com/paper/1511.05049/) (2015)
- [Facial Landmarks Localization using Cascaded Neural Networks](https://mlsanity.com/paper/1805.01760/) (2018)
- [Robust Facial Landmark Detection via Aggregation on Geometrically Manipulated Faces](https://mlsanity.com/paper/2001.03113/) (2020)

## Built on by (most cited first, 6 of 6)

- [ACR Loss: Adaptive Coordinate-based Regression Loss for Face Alignment](https://mlsanity.com/paper/2203.15835/) (2022)
- [XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks](https://mlsanity.com/paper/2111.10854/) (2021)
- [A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection](https://mlsanity.com/paper/2410.19898/) (2024)
- [Linguistic-Based Mild Cognitive Impairment Detection Using Informative Loss](https://mlsanity.com/paper/2402.01690/) (2024)
- [GANalyzer: Analysis and Manipulation of GANs Latent Space for Controllable Face Synthesis](https://mlsanity.com/paper/2302.00908/) (2023)
- [Improving Facial Landmark Detection Accuracy and Efficiency with Knowledge Distillation](https://mlsanity.com/paper/2404.06029/) (2024)

## Abstract (arXiv)

Facial landmark detection is a vital step for numerous facial image analysis applications. Although some deep learning-based methods have achieved good performances in this task, they are often not suitable for running on mobile devices. Such methods rely on networks with many parameters, which makes the training and inference time-consuming. Training lightweight neural networks such as MobileNets are often challenging, and the models might have low accuracy. Inspired by knowledge distillation (KD), this paper presents a novel loss function to train a lightweight Student network (e.g., MobileNetV2) for facial landmark detection. We use two Teacher networks, a Tolerant-Teacher and a Tough-Teacher in conjunction with the Student network. The Tolerant-Teacher is trained using Soft-landmarks created by active shape models, while the Tough-Teacher is trained using the ground truth (aka Hard-landmarks) landmark points. To utilize the facial landmark points predicted by the Teacher networks, we define an Assistive Loss (ALoss) for each Teacher network. Moreover, we define a loss function called KD-Loss that utilizes the facial landmark points predicted by the two pre-trained Teacher networks (EfficientNet-b3) to guide the lightweight Student network towards predicting the Hard-landmarks. Our experimental results on three challenging facial datasets show that the proposed architecture will result in a better-trained Student network that can extract facial landmark points with high accuracy.