# Boosting of Head Pose Estimation by Knowledge Distillation

arXiv: https://arxiv.org/abs/2108.09183 · HTML: https://mlsanity.com/paper/2108.09183/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 80 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition](https://mlsanity.com/paper/1603.01249/) (2016)
- [Facial Landmark Detection: a Literature Survey](https://mlsanity.com/paper/1805.05563/) (2018)
- [Ensemble Knowledge Distillation for Learning Improved and Efficient Networks](https://mlsanity.com/paper/1909.08097/) (2019)
- [A Vector-based Representation to Enhance Head Pose Estimation](https://mlsanity.com/paper/2010.07184/) (2020)
- [Pose-Robust Face Recognition via Deep Residual Equivariant Mapping](https://mlsanity.com/paper/1803.00839/) (2018)
- [FEED: Feature-level Ensemble for Knowledge Distillation](https://mlsanity.com/paper/1909.10754/) (2019)
- [FacePoseNet: Making a Case for Landmark-Free Face Alignment](https://mlsanity.com/paper/1708.07517/) (2017)
- [Improving Head Pose Estimation with a Combined Loss and Bounding Box Margin Adjustment](https://mlsanity.com/paper/1905.08609/) (2019)
- [Nose, eyes and ears: Head pose estimation by locating facial keypoints](https://mlsanity.com/paper/1812.00739/) (2018)
- [An Information-Theoretic View for Deep Learning](https://mlsanity.com/paper/1804.09060/) (2018)
- [Face-from-Depth for Head Pose Estimation on Depth Images](https://mlsanity.com/paper/1712.05277/) (2017)
- [Single Image based Head Pose Estimation with Spherical Parameterization and 3D Morphing](https://mlsanity.com/paper/1907.09217/) (2019)
- [Hybrid coarse-fine classification for head pose estimation](https://mlsanity.com/paper/1901.06778/) (2019)
- [From Depth Data to Head Pose Estimation: a Siamese approach](https://mlsanity.com/paper/1703.03624/) (2017)

## Abstract (arXiv)

We propose a response-based method of knowledge distillation (KD) for the head pose estimation problem. A student model trained by the proposed KD achieves results better than a teacher model, which is atypical for the response-based method. Our method consists of two stages. In the first stage, we trained the base neural network (NN), which has one regression head and four regression via classification (RvC) heads. We build the convolutional ensemble over the base NN using offsets of face bounding boxes over a regular grid. In the second stage, we perform KD from the convolutional ensemble into the final NN with one RvC head. The KD improves the results by an average of 7.7% compared to base NN. This feature makes it possible to use KD as a booster and effectively train deeper NNs. NNs trained by our KD method partially improved the state-of-the-art results. KD-ResNet152 has the best results, and KD-ResNet18 has a better result on the AFLW2000 dataset than any previous method.We have made publicly available trained NNs and face bounding boxes for the 300W-LP, AFLW, AFLW2000, and BIWI datasets.Our method potentially can be effective for other regression problems.