# EmojiHeroVR: A Study on Facial Expression Recognition under Partial Occlusion from Head-Mounted Displays

arXiv: https://arxiv.org/abs/2410.03331 · HTML: https://mlsanity.com/paper/2410.03331/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [From Facial Expression Recognition to Interpersonal Relation Prediction](https://mlsanity.com/paper/1609.06426/) (2016)
- [Facial expression and attributes recognition based on multi-task learning of lightweight neural networks](https://mlsanity.com/paper/2103.17107/) (2021)
- [Teacher-Student Training and Triplet Loss for Facial Expression Recognition under Occlusion](https://mlsanity.com/paper/2008.01003/) (2020)
- [Virtual Reality for Emotion Elicitation -- A Review](https://mlsanity.com/paper/2111.04461/) (2021)
- [Teacher-Student Training and Triplet Loss to Reduce the Effect of Drastic Face Occlusion](https://mlsanity.com/paper/2111.10561/) (2021)
- [Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning](https://mlsanity.com/paper/2008.05563/) (2020)
- [Recognizing Facial Expressions of Occluded Faces using Convolutional Neural Networks](https://mlsanity.com/paper/1911.04852/) (2019)

## Built on by (most cited first, 2 of 2)

- [Unimodal and Multimodal Static Facial Expression Recognition for Virtual Reality Users with EmoHeVRDB](https://mlsanity.com/paper/2412.11306/) (2024)
- [Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG](https://mlsanity.com/paper/2609.03569/) (2026)

## Abstract (arXiv)

Emotion recognition promotes the evaluation and enhancement of Virtual Reality (VR) experiences by providing emotional feedback and enabling advanced personalization. However, facial expressions are rarely used to recognize users' emotions, as Head-Mounted Displays (HMDs) occlude the upper half of the face. To address this issue, we conducted a study with 37 participants who played our novel affective VR game EmojiHeroVR. The collected database, EmoHeVRDB (EmojiHeroVR Database), includes 3,556 labeled facial images of 1,778 reenacted emotions. For each labeled image, we also provide 29 additional frames recorded directly before and after the labeled image to facilitate dynamic Facial Expression Recognition (FER). Additionally, EmoHeVRDB includes data on the activations of 63 facial expressions captured via the Meta Quest Pro VR headset for each frame. Leveraging our database, we conducted a baseline evaluation on the static FER classification task with six basic emotions and neutral using the EfficientNet-B0 architecture. The best model achieved an accuracy of 69.84% on the test set, indicating that FER under HMD occlusion is feasible but significantly more challenging than conventional FER.