Democratizing the Creation of Animatable Facial Avatars
Not on the map. Why?
Abstract (arXiv)
In high-end visual effects pipelines, a customized (and expensive) light stage system is (typically) used to scan an actor in order to acquire both geometry and texture for various expressions. Aiming towards democratization, we propose a novel pipeline for obtaining geometry and texture as well as enough expression information to build a customized person-specific animation rig without using a light stage or any other high-end hardware (or manual cleanup). A key novel idea consists of warping real-world images to align with the geometry of a template avatar and subsequently projecting the warped image into the template avatar's texture; importantly, this allows us to leverage baked-in real-world lighting/texture information in order to create surrogate facial features (and bridge the domain gap) for the sake of geometry reconstruction. Not only can our method be used to obtain a neutral expression geometry and de-lit texture, but it can also be used to improve avatars after they have been imported into an animation system (noting that such imports tend to be lossy, while also hallucinating various features). Since a default animation rig will contain template expressions that do not correctly correspond to those of a particular individual, we use a Simon Says approach to capture various expressions and build a person-specific animation rig (that moves like they do). Our aforementioned warping/projection method has high enough efficacy to reconstruct geometry corresponding to each expressions.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 105 references that are in the map. First posted on arXiv 29 Jan 2024.
Builds on
38 of 105 references.
References · 38
- 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
- 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
- 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
- 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
- 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
- 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
- 2018Nonlinear 3D Face Morphable Modelcited by 157
- 2022Reconstructing Personalized Semantic Facial NeRF Models From Monocular Videocited by 128
- 2022Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothingcited by 126
- 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
- 2022FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Datasetcited by 98
- 2018FML: Face Model Learning from Videoscited by 78
26 more
- 2020Learning Compositional Radiance Fields of Dynamic Human Headscited by 78
- 2017Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regressioncited by 66
- 20163D Face Reconstruction by Learning from Synthetic Datacited by 64
- 2020A Morphable Face Albedo Modelcited by 54
- 2017End-to-end 3D face reconstruction with deep neural networkscited by 47
- 2019MVF-Net: Multi-View 3D Face Morphable Model Regressioncited by 45
- 2022FDNeRF: Few-shot Dynamic Neural Radiance Fields for Face Reconstruction and Expression Editingcited by 38
- 2023Instant Multi-View Head Capture through Learnable Registrationcited by 34
- 2022Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguationcited by 32
- 2022State of the Art in Dense Monocular Non-Rigid 3D Reconstructioncited by 30
- 2020Personalized Face Modeling for Improved Face Reconstruction and Motion Retargetingcited by 27
- 2023Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wildcited by 25
- 2023LPFF: A Portrait Dataset for Face Generators Across Large Posescited by 22
- 2021A Light Stage on Every Deskcited by 21
- 2022High-Res Facial Appearance Capture from Polarized Smartphone Imagescited by 21
- 2020Fast and Robust Face-to-Parameter Translation for Game Character Auto-Creationcited by 19
- 2022Video-driven Neural Physically-based Facial Asset for Productioncited by 17
- 2022S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Imagecited by 17
- 2021MeInGame: Create a Game Character Face from a Single Portraitcited by 16
- 2020Relightable 3D Head Portraits from a Smartphone Videocited by 8
- 2021Unsupervised High-Fidelity Facial Texture Generation and Reconstructioncited by 6
- 2016A Comprehensive Performance Evaluation of Deformable Face Tracking "In-the-Wild"cited by 5
- 2018Head Reconstruction from Internet Photoscited by 5
- 2021Implicit Neural Deformation for Sparse-View Face Reconstructioncited by 4
- 2022Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelinescited by 4
- 2018Deep Energies for Estimating Three-Dimensional Facial Pose and Expressioncited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.