# Self-Supervised 3D Face Reconstruction via Conditional Estimation

arXiv: https://arxiv.org/abs/2110.04800 · HTML: https://mlsanity.com/paper/2110.04800/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 10 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [FML: Face Model Learning from Videos](https://mlsanity.com/paper/1812.07603/) (2018)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression](https://mlsanity.com/paper/1703.07834/) (2017)
- [Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation](https://mlsanity.com/paper/1703.10131/) (2017)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)
- [3D Face Morphable Models "In-the-Wild"](https://mlsanity.com/paper/1701.05360/) (2017)
- [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018)
- [Dense Face Alignment](https://mlsanity.com/paper/1709.01442/) (2017)
- [Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders](https://mlsanity.com/paper/1904.03525/) (2019)
- [3D Dense Face Alignment via Graph Convolution Networks](https://mlsanity.com/paper/1904.05562/) (2019)

## Built on by (most cited first, 17 of 17)

- [Learning Disentangled Avatars with Hybrid 3D Representations](https://mlsanity.com/paper/2309.06441/) (2023)
- [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://mlsanity.com/paper/2203.09729/) (2022)
- [Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation](https://mlsanity.com/paper/2205.03962/) (2022)
- [Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry](https://mlsanity.com/paper/2110.09772/) (2021)
- [H3WB: Human3.6M 3D WholeBody Dataset and Benchmark](https://mlsanity.com/paper/2211.15692/) (2022)
- [HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details](https://mlsanity.com/paper/2303.11225/) (2023)
- [Structural Causal 3D Reconstruction](https://mlsanity.com/paper/2207.10156/) (2022)
- [Inverse Rendering of Translucent Objects using Physical and Neural Renderers](https://mlsanity.com/paper/2305.08336/) (2023)
- [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024)
- [Universal Facial Encoding of Codec Avatars from VR Headsets](https://mlsanity.com/paper/2407.13038/) (2024)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignment](https://mlsanity.com/paper/2305.11522/) (2023)
- [Monocular Identity-Conditioned Facial Reflectance Reconstruction](https://mlsanity.com/paper/2404.00301/) (2024)
- [PIFu for the Real World: A Self-supervised Framework to Reconstruct Dressed Human from Single-view Images](https://mlsanity.com/paper/2208.10769/) (2022)
- [High-Fidelity Facial Albedo Estimation via Texture Quantization](https://mlsanity.com/paper/2406.13149/) (2024)
- [High-Quality Facial Albedo Generation for 3D Face Reconstruction from a Single Image using a Coarse-to-Fine Approach](https://mlsanity.com/paper/2506.13233/) (2025)
- [XYZFlow:Scaling Multi dimensional Shortcut Flows for Efficient Generative Modeling](https://mlsanity.com/paper/2608.12276/) (2026)

## Abstract (arXiv)

We present a conditional estimation (CEST) framework to learn 3D facial parameters from 2D single-view images by self-supervised training from videos. CEST is based on the process of analysis by synthesis, where the 3D facial parameters (shape, reflectance, viewpoint, and illumination) are estimated from the face image, and then recombined to reconstruct the 2D face image. In order to learn semantically meaningful 3D facial parameters without explicit access to their labels, CEST couples the estimation of different 3D facial parameters by taking their statistical dependency into account. Specifically, the estimation of any 3D facial parameter is not only conditioned on the given image, but also on the facial parameters that have already been derived. Moreover, the reflectance symmetry and consistency among the video frames are adopted to improve the disentanglement of facial parameters. Together with a novel strategy for incorporating the reflectance symmetry and consistency, CEST can be efficiently trained with in-the-wild video clips. Both qualitative and quantitative experiments demonstrate the effectiveness of CEST.