# Personalized Face Modeling for Improved Face Reconstruction and Motion Retargeting

arXiv: https://arxiv.org/abs/2007.06759 · HTML: https://mlsanity.com/paper/2007.06759/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 27 of 27)

- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video](https://mlsanity.com/paper/2210.06108/) (2022)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [3D Gaussian Blendshapes for Head Avatar Animation](https://mlsanity.com/paper/2404.19398/) (2024)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2304.01436/) (2023)
- [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://mlsanity.com/paper/2203.09729/) (2022)
- [Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation](https://mlsanity.com/paper/2205.03962/) (2022)
- [Riggable 3D Face Reconstruction via In-Network Optimization](https://mlsanity.com/paper/2104.03493/) (2021)
- [RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars](https://mlsanity.com/paper/2503.12886/) (2025)
- [Learning a 3D Morphable Face Reflectance Model from Low-cost Data](https://mlsanity.com/paper/2303.11686/) (2023)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual Reality](https://mlsanity.com/paper/2104.04794/) (2021)
- [Deep Face Video Inpainting via UV Mapping](https://mlsanity.com/paper/2109.00681/) (2021)
- [High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering](https://mlsanity.com/paper/2401.08398/) (2024)
- [Bring Your Own Character: A Holistic Solution for Automatic Facial Animation Generation of Customized Characters](https://mlsanity.com/paper/2402.13724/) (2024)
- [AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction](https://mlsanity.com/paper/2304.13115/) (2023)
- [MagicMirror: Fast and High-Quality Avatar Generation with a Constrained Search Space](https://mlsanity.com/paper/2404.01296/) (2024)
- [High-fidelity Interpretable Inverse Rig: An Accurate and Sparse Solution Optimizing the Quartic Blendshape Model](https://mlsanity.com/paper/2302.04820/) (2023)
- [Facial Expression Re-targeting from a Single Character](https://mlsanity.com/paper/2306.12188/) (2023)
- [Democratizing the Creation of Animatable Facial Avatars](https://mlsanity.com/paper/2401.16534/) (2024)
- [LightAvatar: Efficient Head Avatar as Dynamic Neural Light Field](https://mlsanity.com/paper/2409.18057/) (2024)
- [IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos](https://mlsanity.com/paper/2504.19165/) (2025)
- [Out-of-distribution Few-shot Learning For Edge Devices without Model Fine-tuning](https://mlsanity.com/paper/2304.06309/) (2023)
- [TimeWalker: Personalized Neural Space for Lifelong Head Avatars](https://mlsanity.com/paper/2412.02421/) (2024)
- [MGAvatar: Mesh-Bound Gaussians for Head Avatar Geometry and Appearance Modeling](https://mlsanity.com/paper/2609.12850/) (2026)

## Abstract (arXiv)

Traditional methods for image-based 3D face reconstruction and facial motion retargeting fit a 3D morphable model (3DMM) to the face, which has limited modeling capacity and fail to generalize well to in-the-wild data. Use of deformation transfer or multilinear tensor as a personalized 3DMM for blendshape interpolation does not address the fact that facial expressions result in different local and global skin deformations in different persons. Moreover, existing methods learn a single albedo per user which is not enough to capture the expression-specific skin reflectance variations. We propose an end-to-end framework that jointly learns a personalized face model per user and per-frame facial motion parameters from a large corpus of in-the-wild videos of user expressions. Specifically, we learn user-specific expression blendshapes and dynamic (expression-specific) albedo maps by predicting personalized corrections on top of a 3DMM prior. We introduce novel constraints to ensure that the corrected blendshapes retain their semantic meanings and the reconstructed geometry is disentangled from the albedo. Experimental results show that our personalization accurately captures fine-grained facial dynamics in a wide range of conditions and efficiently decouples the learned face model from facial motion, resulting in more accurate face reconstruction and facial motion retargeting compared to state-of-the-art methods.