# Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual Reality arXiv: https://arxiv.org/abs/2104.04794 ยท HTML: https://mlsanity.com/paper/2104.04794/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 51 references that are in the map. - First posted on arXiv 10 Apr 2021. ## Builds on (most cited first, 27 of 27) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018) - [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016) - [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020) - [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015) - [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020) - [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014) - [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020) - [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019) - [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018) - [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019) - [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018) - [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019) - [FML: Face Model Learning from Videos](https://mlsanity.com/paper/1812.07603/) (2018) - [Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression](https://mlsanity.com/paper/1703.07834/) (2017) - [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018) - [Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks](https://mlsanity.com/paper/2003.05653/) (2020) - [Audio- and Gaze-driven Facial Animation of Codec Avatars](https://mlsanity.com/paper/2008.05023/) (2020) - [Personalized Face Modeling for Improved Face Reconstruction and Motion Retargeting](https://mlsanity.com/paper/2007.06759/) (2020) - [Joint Face Detection and Facial Motion Retargeting for Multiple Faces](https://mlsanity.com/paper/1902.10744/) (2019) - [Expressive Telepresence via Modular Codec Avatars](https://mlsanity.com/paper/2008.11789/) (2020) - [High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation](https://mlsanity.com/paper/2103.15876/) (2021) - [Eyemotion: Classifying facial expressions in VR using eye-tracking cameras](https://mlsanity.com/paper/1707.07204/) (2017) - [Pose-Invariant Face Alignment with a Single CNN](https://mlsanity.com/paper/1707.06286/) (2017) - [EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment](https://mlsanity.com/paper/1905.10822/) (2019) - [Unsupervised Learning Facial Parameter Regressor for Action Unit Intensity Estimation via Differentiable Renderer](https://mlsanity.com/paper/2008.08862/) (2020) ## Built on by (most cited first, 8 of 8) - [Benchmarking Micro-action Recognition: Dataset, Methods, and Applications](https://mlsanity.com/paper/2403.05234/) (2024) - [SingingHead: A Large-scale 4D Dataset for Singing Head Animation](https://mlsanity.com/paper/2312.04369/) (2023) - [EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars](https://mlsanity.com/paper/2410.01835/) (2024) - [Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelines](https://mlsanity.com/paper/2204.10746/) (2022) - [INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields](https://mlsanity.com/paper/2312.16197/) (2023) - [GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images](https://mlsanity.com/paper/2308.03413/) (2023) - [EgoAnimate: Generating Human Animations from Egocentric top-down Views](https://mlsanity.com/paper/2507.09230/) (2025) - [Enhancing Facial Expression Recognition in Head-Mounted Displays with Synthetic Data](https://mlsanity.com/paper/2607.04490/) (2026) ## Abstract (arXiv) Social presence, the feeling of being there with a real person, will fuel the next generation of communication systems driven by digital humans in virtual reality (VR). The best 3D video-realistic VR avatars that minimize the uncanny effect rely on person-specific (PS) models. However, these PS models are time-consuming to build and are typically trained with limited data variability, which results in poor generalization and robustness. Major sources of variability that affects the accuracy of facial expression transfer algorithms include using different VR headsets (e.g., camera configuration, slop of the headset), facial appearance changes over time (e.g., beard, make-up), and environmental factors (e.g., lighting, backgrounds). This is a major drawback for the scalability of these models in VR. This paper makes progress in overcoming these limitations by proposing an end-to-end multi-identity architecture (MIA) trained with specialized augmentation strategies. MIA drives the shape component of the avatar from three cameras in the VR headset (two eyes, one mouth), in untrained subjects, using minimal personalized information (i.e., neutral 3D mesh shape). Similarly, if the PS texture decoder is available, MIA is able to drive the full avatar (shape+texture) robustly outperforming PS models in challenging scenarios. Our key contribution to improve robustness and generalization, is that our method implicitly decouples, in an unsupervised manner, the facial expression from nuisance factors (e.g., headset, environment, facial appearance). We demonstrate the superior performance and robustness of the proposed method versus state-of-the-art PS approaches in a variety of experiments.