# Unsupervised Depth Estimation, 3D Face Rotation and Replacement

arXiv: https://arxiv.org/abs/1803.09202 · HTML: https://mlsanity.com/paper/1803.09202/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2018.

## Built on by (most cited first, 10 of 10)

- [LOLNeRF: Learn from One Look](https://mlsanity.com/paper/2111.09996/) (2021)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [De-rendering 3D Objects in the Wild](https://mlsanity.com/paper/2201.02279/) (2022)
- [Face Generation and Editing with StyleGAN: A Survey](https://mlsanity.com/paper/2212.09102/) (2022)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detection](https://mlsanity.com/paper/2212.05667/) (2022)
- [Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a Single Image](https://mlsanity.com/paper/2210.15200/) (2022)
- [Unsupervised Style-based Explicit 3D Face Reconstruction from Single Image](https://mlsanity.com/paper/2304.12455/) (2023)
- [Illumination and Shadows in Head Rotation: experiments with Denoising Diffusion Models](https://mlsanity.com/paper/2308.06057/) (2023)

## Abstract (arXiv)

We present an unsupervised approach for learning to estimate three dimensional (3D) facial structure from a single image while also predicting 3D viewpoint transformations that match a desired pose and facial geometry. We achieve this by inferring the depth of facial keypoints of an input image in an unsupervised manner, without using any form of ground-truth depth information. We show how it is possible to use these depths as intermediate computations within a new backpropable loss to predict the parameters of a 3D affine transformation matrix that maps inferred 3D keypoints of an input face to the corresponding 2D keypoints on a desired target facial geometry or pose. Our resulting approach, called DepthNets, can therefore be used to infer plausible 3D transformations from one face pose to another, allowing faces to be frontalized, transformed into 3D models or even warped to another pose and facial geometry. Lastly, we identify certain shortcomings with our formulation, and explore adversarial image translation techniques as a post-processing step to re-synthesize complete head shots for faces re-targeted to different poses or identities.