# Hand2Face: Automatic Synthesis and Recognition of Hand Over Face Occlusions

arXiv: https://arxiv.org/abs/1708.00370 · HTML: https://mlsanity.com/paper/1708.00370/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2017.

## Built on by (most cited first, 8 of 8)

- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Synthetic Data in Human Analysis: A Survey](https://mlsanity.com/paper/2208.09191/) (2022)
- [Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets](https://mlsanity.com/paper/2205.06218/) (2022)
- [VGGHeads: 3D Multi Head Alignment with a Large-Scale Synthetic Dataset](https://mlsanity.com/paper/2407.18245/) (2024)
- [Self-Supervised Robustifying Guidance for Monocular 3D Face Reconstruction](https://mlsanity.com/paper/2112.14382/) (2021)
- [Towards Inclusive HRI: Using Sim2Real to Address Underrepresentation in Emotion Expression Recognition](https://mlsanity.com/paper/2208.07472/) (2022)
- [Occlusion Aware Student Emotion Recognition based on Facial Action Unit Detection](https://mlsanity.com/paper/2307.09465/) (2023)
- [FaceTouch: Detecting hand-to-face touch with supervised contrastive learning to assist in tracing infectious disease](https://mlsanity.com/paper/2308.12840/) (2023)

## Abstract (arXiv)

A person's face discloses important information about their affective state. Although there has been extensive research on recognition of facial expressions, the performance of existing approaches is challenged by facial occlusions. Facial occlusions are often treated as noise and discarded in recognition of affective states. However, hand over face occlusions can provide additional information for recognition of some affective states such as curiosity, frustration and boredom. One of the reasons that this problem has not gained attention is the lack of naturalistic occluded faces that contain hand over face occlusions as well as other types of occlusions. Traditional approaches for obtaining affective data are time demanding and expensive, which limits researchers in affective computing to work on small datasets. This limitation affects the generalizability of models and deprives researchers from taking advantage of recent advances in deep learning that have shown great success in many fields but require large volumes of data. In this paper, we first introduce a novel framework for synthesizing naturalistic facial occlusions from an initial dataset of non-occluded faces and separate images of hands, reducing the costly process of data collection and annotation. We then propose a model for facial occlusion type recognition to differentiate between hand over face occlusions and other types of occlusions such as scarves, hair, glasses and objects. Finally, we present a model to localize hand over face occlusions and identify the occluded regions of the face.