# FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

arXiv: https://arxiv.org/abs/1610.03151 · HTML: https://mlsanity.com/paper/1610.03151/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2016.

## Built on by (most cited first, 12 of 12)

- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videos](https://mlsanity.com/paper/2207.11094/) (2022)
- [Tri2-plane: Thinking Head Avatar via Feature Pyramid](https://mlsanity.com/paper/2401.09386/) (2024)
- [EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes](https://mlsanity.com/paper/2206.08428/) (2022)
- [EgoRenderer: Rendering Human Avatars from Egocentric Camera Images](https://mlsanity.com/paper/2111.12685/) (2021)
- [SPARK: Self-supervised Personalized Real-time Monocular Face Capture](https://mlsanity.com/paper/2409.07984/) (2024)
- [VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction](https://mlsanity.com/paper/2409.09740/) (2024)
- [Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelines](https://mlsanity.com/paper/2204.10746/) (2022)
- [Attention based Occlusion Removal for Hybrid Telepresence Systems](https://mlsanity.com/paper/2112.01098/) (2021)
- [Expression-aware video inpainting for HMD removal in XR applications](https://mlsanity.com/paper/2401.14136/) (2024)
- [Geometry-Aware Video Inpainting for Joint Headset Occlusion Removal and Face Reconstruction in Social XR](https://mlsanity.com/paper/2508.12336/) (2025)
- [Discovering Geometric Biases in 3D Face Reconstruction: A Curvature-Aware Spectral Framework for Fairness Evaluation](https://mlsanity.com/paper/2607.07486/) (2026)

## Abstract (arXiv)

We propose FaceVR, a novel image-based method that enables video teleconferencing in VR based on self-reenactment. State-of-the-art face tracking methods in the VR context are focused on the animation of rigged 3d avatars. While they achieve good tracking performance the results look cartoonish and not real. In contrast to these model-based approaches, FaceVR enables VR teleconferencing using an image-based technique that results in nearly photo-realistic outputs. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD), as well as a new data-driven approach for eye tracking from monocular videos. Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances. For instance, we can alter facial expressions or change gaze directions in the prerecorded target video. In a live setup, we apply these newly-introduced algorithmic components.