# Enhancing Facial Expression Recognition in Head-Mounted Displays with Synthetic Data

arXiv: https://arxiv.org/abs/2607.04490 · HTML: https://mlsanity.com/paper/2607.04490/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 69 references that are in the map.
- First posted on arXiv 5 Jul 2026.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [Infinite Photorealistic Worlds using Procedural Generation](https://mlsanity.com/paper/2306.09310/) (2023)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [Synthetic Data for Deep Learning](https://mlsanity.com/paper/1909.11512/) (2019)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [GPAvatar: Generalizable and Precise Head Avatar from Image(s)](https://mlsanity.com/paper/2401.10215/) (2024)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition](https://mlsanity.com/paper/2207.10299/) (2022)
- [High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies](https://mlsanity.com/paper/2010.05562/) (2020)
- [Photo-Realistic Facial Details Synthesis from Single Image](https://mlsanity.com/paper/1903.10873/) (2019)
- [FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction](https://mlsanity.com/paper/2211.13874/) (2022)

## Abstract (arXiv)

Facial expression recognition (FER) is crucial for social interaction in mixed reality environments that employ head-mounted displays (HMD). However, collecting FER data from head-mounted cameras (HMC) is challenging due to privacy concerns and the diversity of HMD platforms. Moreover, existing FER datasets are not directly applicable due to the unique perspectives of HMCs. The lack of sufficient data hinders the development of neural network-based HMC FER methods. To address data scarcity, we propose a data synthesis framework that generates HMC-view images from frontal-view images, leveraging abundant existing annotated datasets. Specifically, we first reconstruct 3D textured meshes from images and then apply a configurable camera system to render images from the HMC perspective. Additionally, we introduce a texture-space alignment network (TSAN) that enables accurate texture sampling from images to preserve detailed facial expressions. To evaluate the proposed method, we conduct extensive experiments on both simulated and real HMC datasets. Experimental results demonstrate that models trained on our synthetic dataset outperform those trained on existing datasets and exhibit better generalization across different camera configurations.