# HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details

arXiv: https://arxiv.org/abs/2303.11225 · HTML: https://mlsanity.com/paper/2303.11225/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 85 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 56)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset](https://mlsanity.com/paper/2203.14057/) (2022)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)

## Built on by (most cited first, 22 of 22)

- [GAIA: Zero-shot Talking Avatar Generation](https://mlsanity.com/paper/2311.15230/) (2023)
- [DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars](https://mlsanity.com/paper/2311.18635/) (2023)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow](https://mlsanity.com/paper/2404.09819/) (2024)
- [MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading](https://mlsanity.com/paper/2312.13091/) (2023)
- [ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment](https://mlsanity.com/paper/2308.14448/) (2023)
- [DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image](https://mlsanity.com/paper/2406.17988/) (2024)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction](https://mlsanity.com/paper/2409.09740/) (2024)
- [FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models](https://mlsanity.com/paper/2312.04465/) (2023)
- [S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch](https://mlsanity.com/paper/2408.01218/) (2024)
- [SEREP: Semantic Facial Expression Representation for Robust In-the-Wild Capture and Retargeting](https://mlsanity.com/paper/2412.14371/) (2024)
- [AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars](https://mlsanity.com/paper/2507.02419/) (2025)
- [GeoAvatar: Adaptive Geometrical Gaussian Splatting for 3D Head Avatar](https://mlsanity.com/paper/2507.18155/) (2025)
- [High-Quality Facial Albedo Generation for 3D Face Reconstruction from a Single Image using a Coarse-to-Fine Approach](https://mlsanity.com/paper/2506.13233/) (2025)
- [GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance](https://mlsanity.com/paper/2312.07385/) (2023)
- [Skullptor: High Fidelity 3D Head Reconstruction in Seconds with Multi-View Normal Prediction](https://mlsanity.com/paper/2602.21100/) (2026)
- [Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading](https://mlsanity.com/paper/2606.28144/) (2026)
- [JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync](https://mlsanity.com/paper/2507.20452/) (2025)
- [Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture](https://mlsanity.com/paper/2512.16397/) (2025)
- [SelfieAvatar: Real-time Head Avatar reenactment from a Selfie Video](https://mlsanity.com/paper/2601.18851/) (2026)
- [Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization](https://mlsanity.com/paper/2602.00639/) (2026)

## Abstract (arXiv)

3D Morphable Models (3DMMs) demonstrate great potential for reconstructing faithful and animatable 3D facial surfaces from a single image. The facial surface is influenced by the coarse shape, as well as the static detail (e,g., person-specific appearance) and dynamic detail (e.g., expression-driven wrinkles). Previous work struggles to decouple the static and dynamic details through image-level supervision, leading to reconstructions that are not realistic. In this paper, we aim at high-fidelity 3D face reconstruction and propose HiFace to explicitly model the static and dynamic details. Specifically, the static detail is modeled as the linear combination of a displacement basis, while the dynamic detail is modeled as the linear interpolation of two displacement maps with polarized expressions. We exploit several loss functions to jointly learn the coarse shape and fine details with both synthetic and real-world datasets, which enable HiFace to reconstruct high-fidelity 3D shapes with animatable details. Extensive quantitative and qualitative experiments demonstrate that HiFace presents state-of-the-art reconstruction quality and faithfully recovers both the static and dynamic details. Our project page can be found at https://project-hiface.github.io.