mlsanityData to 7 Oct 2026

REFA: Real-time Egocentric Facial Animations for Virtual Reality

arXiv 2601.03507 ↗First posted 7 Jan 2026cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a novel system for real-time tracking of facial expressions using egocentric views captured from a set of infrared cameras embedded in a virtual reality (VR) headset. Our technology facilitates any user to accurately drive the facial expressions of virtual characters in a non-intrusive manner and without the need of a lengthy calibration step. At the core of our system is a distillation based approach to train a machine learning model on heterogeneous data and labels coming form multiple sources, eg synthetic and real images. As part of our dataset, we collected 18k diverse subjects using a lightweight capture setup consisting of a mobile phone and a custom VR headset with extra cameras. To process this data, we developed a robust differentiable rendering pipeline enabling us to automatically extract facial expression labels. Our system opens up new avenues for communication and expression in virtual environments, with applications in video conferencing, gaming, entertainment, and remote collaboration.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 54 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

25 of 54 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017Mask R-CNNcited by 7,259
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  9. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  10. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  11. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  12. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
13 more
  1. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  2. 2018Deep Appearance Models for Face Renderingcited by 164
  3. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  4. 2020DeepFaceLab: Integrated, flexible and extensible face-swapping frameworkcited by 129
  5. 2022RigNeRF: Fully Controllable Neural 3D Portraitscited by 126
  6. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  7. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidancecited by 77
  8. 20163D Face Reconstruction by Learning from Synthetic Datacited by 64
  9. 2020High-Fidelity 3D Digital Human Head Creation from RGB-D Selfiescited by 47
  10. 2019MVF-Net: Multi-View 3D Face Morphable Model Regressioncited by 45
  11. 2019Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networkscited by 34
  12. 2021Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstructioncited by 33
  13. 2016Real-Time Facial Segmentation and Performance Capture from RGB Inputcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMGcited by 0