# Headset: Human emotion awareness under partial occlusions multimodal dataset

arXiv: https://arxiv.org/abs/2402.09107 · HTML: https://mlsanity.com/paper/2402.09107/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 65 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018)
- [Point Cloud Quality Assessment: Dataset Construction and Learning-based No-Reference Metric](https://mlsanity.com/paper/2012.11895/) (2020)
- [Facial expression and attributes recognition based on multi-task learning of lightweight neural networks](https://mlsanity.com/paper/2103.17107/) (2021)
- [3D Human Body Reconstruction from a Single Image via Volumetric Regression](https://mlsanity.com/paper/1809.03770/) (2018)
- [HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media](https://mlsanity.com/paper/2110.07235/) (2021)
- [An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing](https://mlsanity.com/paper/1712.03084/) (2017)
- [Identity Preserving Face Completion for Large Ocular Region Occlusion](https://mlsanity.com/paper/1807.08772/) (2018)
- [CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routing](https://mlsanity.com/paper/2101.03503/) (2021)

## Built on by (most cited first, 4 of 4)

- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR](https://mlsanity.com/paper/2506.05683/) (2025)
- [3D2M Dataset: A 3-Dimension diverse Mesh Dataset](https://mlsanity.com/paper/2410.07415/) (2024)
- [Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG](https://mlsanity.com/paper/2609.03569/) (2026)

## Abstract (arXiv)

The volumetric representation of human interactions is one of the fundamental domains in the development of immersive media productions and telecommunication applications. Particularly in the context of the rapid advancement of Extended Reality (XR) applications, this volumetric data has proven to be an essential technology for future XR elaboration. In this work, we present a new multimodal database to help advance the development of immersive technologies. Our proposed database provides ethically compliant and diverse volumetric data, in particular 27 participants displaying posed facial expressions and subtle body movements while speaking, plus 11 participants wearing head-mounted displays (HMDs). The recording system consists of a volumetric capture (VoCap) studio, including 31 synchronized modules with 62 RGB cameras and 31 depth cameras. In addition to textured meshes, point clouds, and multi-view RGB-D data, we use one Lytro Illum camera for providing light field (LF) data simultaneously. Finally, we also provide an evaluation of our dataset employment with regard to the tasks of facial expression classification, HMDs removal, and point cloud reconstruction. The dataset can be helpful in the evaluation and performance testing of various XR algorithms, including but not limited to facial expression recognition and reconstruction, facial reenactment, and volumetric video. HEADSET and its all associated raw data and license agreement will be publicly available for research purposes.