# Teacher-Student Training and Triplet Loss to Reduce the Effect of Drastic Face Occlusion arXiv: https://arxiv.org/abs/2111.10561 ยท HTML: https://mlsanity.com/paper/2111.10561/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 33 of its 83 references that are in the map. - First posted on arXiv 20 Nov 2021. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 33) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016) - [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017) - [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013) - [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017) - [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013) - [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015) - [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019) - [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016) - [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020) - [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://mlsanity.com/paper/1405.3531/) (2014) - [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021) - [Deep Learning using Linear Support Vector Machines](https://mlsanity.com/paper/1306.0239/) (2013) - [Learning Metrics from Teachers: Compact Networks for Image Embedding](https://mlsanity.com/paper/1904.03624/) (2019) - [MatConvNet - Convolutional Neural Networks for MATLAB](https://mlsanity.com/paper/1412.4564/) (2014) - [FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition](https://mlsanity.com/paper/1609.06591/) (2016) - [Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networks](https://mlsanity.com/paper/2001.06338/) (2020) - [Local Learning with Deep and Handcrafted Features for Facial Expression Recognition](https://mlsanity.com/paper/1804.10892/) (2018) - [Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks](https://mlsanity.com/paper/1705.07871/) (2017) - [Triplet Distillation for Deep Face Recognition](https://mlsanity.com/paper/1905.04457/) (2019) - [Eyemotion: Classifying facial expressions in VR using eye-tracking cameras](https://mlsanity.com/paper/1707.07204/) (2017) - [Teacher-Student Training and Triplet Loss for Facial Expression Recognition under Occlusion](https://mlsanity.com/paper/2008.01003/) (2020) - [Soft-ranking Label Encoding for Robust Facial Age Estimation](https://mlsanity.com/paper/1906.03625/) (2019) - [Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning](https://mlsanity.com/paper/2008.05563/) (2020) - [Facial Expression Recognition from World Wild Web](https://mlsanity.com/paper/1605.03639/) (2016) ## Built on by (most cited first, 9 of 9) - [Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation](https://mlsanity.com/paper/2211.15597/) (2022) - [Learning Using Generated Privileged Information by Text-to-Image Diffusion Models](https://mlsanity.com/paper/2309.15238/) (2023) - [EmojiHeroVR: A Study on Facial Expression Recognition under Partial Occlusion from Head-Mounted Displays](https://mlsanity.com/paper/2410.03331/) (2024) - [Overcoming Occlusions in the Wild: A Multi-Task Age Head Approach to Age Estimation](https://mlsanity.com/paper/2506.13445/) (2025) - [Masked Student Dataset of Expressions](https://mlsanity.com/paper/2304.03867/) (2023) - [Class Anchor Margin Loss for Content-Based Image Retrieval](https://mlsanity.com/paper/2306.00630/) (2023) - [HB-net: Holistic bursting cell cluster integrated network for occluded multi-objects recognition](https://mlsanity.com/paper/2310.11834/) (2023) - [Unimodal and Multimodal Static Facial Expression Recognition for Virtual Reality Users with EmoHeVRDB](https://mlsanity.com/paper/2412.11306/) (2024) - [DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts](https://mlsanity.com/paper/2608.23363/) (2026) ## Abstract (arXiv) We study a series of recognition tasks in two realistic scenarios requiring the analysis of faces under strong occlusion. On the one hand, we aim to recognize facial expressions of people wearing Virtual Reality (VR) headsets. On the other hand, we aim to estimate the age and identify the gender of people wearing surgical masks. For all these tasks, the common ground is that half of the face is occluded. In this challenging setting, we show that convolutional neural networks (CNNs) trained on fully-visible faces exhibit very low performance levels. While fine-tuning the deep learning models on occluded faces is extremely useful, we show that additional performance gains can be obtained by distilling knowledge from models trained on fully-visible faces. To this end, we study two knowledge distillation methods, one based on teacher-student training and one based on triplet loss. Our main contribution consists in a novel approach for knowledge distillation based on triplet loss, which generalizes across models and tasks. Furthermore, we consider combining distilled models learned through conventional teacher-student training or through our novel teacher-student training based on triplet loss. We provide empirical evidence showing that, in most cases, both individual and combined knowledge distillation methods bring statistically significant performance improvements. We conduct experiments with three different neural models (VGG-f, VGG-face, ResNet-50) on various tasks (facial expression recognition, gender recognition, age estimation), showing consistent improvements regardless of the model or task.