# FaceTouch: Detecting hand-to-face touch with supervised contrastive learning to assist in tracing infectious disease

arXiv: https://arxiv.org/abs/2308.12840 · HTML: https://mlsanity.com/paper/2308.12840/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [RetinaFace: Single-stage Dense Face Localisation in the Wild](https://mlsanity.com/paper/1905.00641/) (2019)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs](https://mlsanity.com/paper/1907.05047/) (2019)
- [Finding Tiny Faces](https://mlsanity.com/paper/1612.04402/) (2016)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [LFFD: A Light and Fast Face Detector for Edge Devices](https://mlsanity.com/paper/1904.10633/) (2019)
- [WeatherNet: Recognising weather and visual conditions from street-level images using deep residual learning](https://mlsanity.com/paper/1910.09910/) (2019)
- [URBAN-i: From urban scenes to mapping slums, transport modes, and pedestrians in cities using deep learning and computer vision](https://mlsanity.com/paper/1809.03609/) (2018)
- [Hand2Face: Automatic Synthesis and Recognition of Hand Over Face Occlusions](https://mlsanity.com/paper/1708.00370/) (2017)
- [CyclingNet: Detecting cycling near misses from video streams in complex urban scenes with deep learning](https://mlsanity.com/paper/2102.00565/) (2021)
- [Towards Interpretable and Robust Hand Detection via Pixel-wise Prediction](https://mlsanity.com/paper/2001.04163/) (2020)
- [LE-HGR: A Lightweight and Efficient RGB-based Online Gesture Recognition Network for Embedded AR Devices](https://mlsanity.com/paper/2001.05654/) (2020)
- [FaceOff: Detecting Face Touching with a Wrist-Worn Accelerometer](https://mlsanity.com/paper/2008.01769/) (2020)
- [ImageSig: A signature transform for ultra-lightweight image recognition](https://mlsanity.com/paper/2205.06929/) (2022)
- [CNN Based Posture-Free Hand Detection](https://mlsanity.com/paper/1809.10432/) (2018)
- [Anchors Based Method for Fingertips Position Estimation from a Monocular RGB Image using Deep Neural Network](https://mlsanity.com/paper/2005.01351/) (2020)
- [Skeleton-aware multi-scale heatmap regression for 2D hand pose estimation](https://mlsanity.com/paper/2105.10904/) (2021)

## Abstract (arXiv)

Through our respiratory system, many viruses and diseases frequently spread and pass from one person to another. Covid-19 served as an example of how crucial it is to track down and cut back on contacts to stop its spread. There is a clear gap in finding automatic methods that can detect hand-to-face contact in complex urban scenes or indoors. In this paper, we introduce a computer vision framework, called FaceTouch, based on deep learning. It comprises deep sub-models to detect humans and analyse their actions. FaceTouch seeks to detect hand-to-face touches in the wild, such as through video chats, bus footage, or CCTV feeds. Despite partial occlusion of faces, the introduced system learns to detect face touches from the RGB representation of a given scene by utilising the representation of the body gestures such as arm movement. This has been demonstrated to be useful in complex urban scenarios beyond simply identifying hand movement and its closeness to faces. Relying on Supervised Contrastive Learning, the introduced model is trained on our collected dataset, given the absence of other benchmark datasets. The framework shows a strong validation in unseen datasets which opens the door for potential deployment.