mlsanityData to 7 Oct 2026

High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation

arXiv 2103.15876 ↗First posted 29 Mar 2021cs.CV eess.IVcited by 13

Not on the map. Why?

Abstract (arXiv)

3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven by video, that can minimize uncanny effects, rely on person-specific models. However, existing person-specific photo-realistic 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This is a major drawback for the scalability of these models in communication systems (e.g., Messenger, Skype, FaceTime) and AR/VR. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar. Extensive experimental validation and comparisons to other state-of-the-art methods demonstrate the effectiveness of the proposed framework in real-world scenarios with variability in pose, expression, and illumination. Please visit https://www.youtube.com/watch?v=dtz1LgZR8cc for more results. Our project page can be found at https://www.cs.rochester.edu/u/lchen63.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

32 of 59 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  3. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  4. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  5. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  6. 2020FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Predictioncited by 196
  7. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  8. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  9. 2018Deep Appearance Models for Face Renderingcited by 164
  10. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  11. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  12. 2018Face Alignment in Full Pose Range: A 3D Total Solutioncited by 147
20 more
  1. 2017Morphable Face Models - An Open Frameworkcited by 138
  2. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  3. 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
  4. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  5. 2019Towards High-fidelity Nonlinear 3D Face Morphable Modelcited by 91
  6. 2019Learning Physics-guided Face Relighting under Directional Lightcited by 90
  7. 2018FML: Face Model Learning from Videoscited by 78
  8. 2020Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistencycited by 78
  9. 2016Fast Optical Flow using Dense Inverse Searchcited by 70
  10. 20163D Face Reconstruction by Learning from Synthetic Datacited by 64
  11. 2017Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translationcited by 56
  12. 2018On Learning 3D Face Morphable Model from In-the-wild Imagescited by 54
  13. 2020Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networkscited by 51
  14. 2020Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Imagescited by 37
  15. 2020Cross-modal Deep Face Normals with Deactivable Skip Connectionscited by 20
  16. 2019Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Trackingcited by 12
  17. 2020Lightweight Photometric Stereo for Facial Details Recoverycited by 8
  18. 2017InverseFaceNet: Deep Monocular Inverse Face Renderingcited by 6
  19. 2018Superimposition-guided Facial Reconstruction from Skullcited by 3
  20. 2016Real-time Halfway Domain Reconstruction of Motion and Geometrycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 2; 2024: 5; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  2. 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
  3. 2022Video-driven Neural Physically-based Facial Asset for Productioncited by 17
  4. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  5. 2021Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual Realitycited by 8
  6. 2023MyStyle++: A Controllable Personalized Generative Priorcited by 8
  7. 2024Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Methodcited by 8
  8. 2024Universal Facial Encoding of Codec Avatars from VR Headsetscited by 7
  9. 2022Force-Aware Interface via Electromyography for Natural VR/AR Interactioncited by 5
  10. 2025DiffListener: Discrete Diffusion Model for Listener Generationcited by 2
  11. 2026ELITE: Efficient Gaussian Head Avatar from a Monocular Video via Learned Initialization and TEst-time Generative Adaptationcited by 1
  12. 2024Fast Registration of Photorealistic Avatars for VR Facial Animationcited by 0
1 more
  1. 2025Generative Head-Mounted Camera Captures for Photorealistic Avatarscited by 0