# Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation

arXiv: https://arxiv.org/abs/2106.09614 · HTML: https://mlsanity.com/paper/2106.09614/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [Training Deep Face Recognition Systems with Synthetic Data](https://mlsanity.com/paper/1802.05891/) (2018)
- [Real-Time Facial Segmentation and Performance Capture from RGB Input](https://mlsanity.com/paper/1604.02647/) (2016)
- [Informed MCMC with Bayesian Neural Networks for Facial Image Analysis](https://mlsanity.com/paper/1811.07969/) (2018)

## Built on by (most cited first, 20 of 20)

- [3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions](https://mlsanity.com/paper/2404.04823/) (2024)
- [DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image](https://mlsanity.com/paper/2406.17988/) (2024)
- [Implicit Shape and Appearance Priors for Few-Shot Full Head Reconstruction](https://mlsanity.com/paper/2310.08784/) (2023)
- [SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians](https://mlsanity.com/paper/2504.12292/) (2025)
- [ASM: Adaptive Skinning Model for High-Quality 3D Face Modeling](https://mlsanity.com/paper/2304.09423/) (2023)
- [FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models](https://mlsanity.com/paper/2312.04465/) (2023)
- [Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection](https://mlsanity.com/paper/2411.07167/) (2024)
- [FaceGPT: Self-supervised Learning to Chat about 3D Human Faces](https://mlsanity.com/paper/2406.07163/) (2024)
- [PIG: Prompt Images Guidance for Night-Time Scene Parsing](https://mlsanity.com/paper/2406.10531/) (2024)
- [MIRACLE3D: Memory-efficient Integrated Robust Approach for Continual Learning on Point Clouds via Shape Model Construction](https://mlsanity.com/paper/2410.06418/) (2024)
- [Approximating Intersections and Differences Between Linear Statistical Shape Models Using Markov Chain Monte Carlo](https://mlsanity.com/paper/2211.16314/) (2022)
- [Learning to Decouple the Lights for 3D Face Texture Modeling](https://mlsanity.com/paper/2412.08524/) (2024)
- [Electromyography-Informed Facial Expression Reconstruction for Physiological-Based Synthesis and Analysis](https://mlsanity.com/paper/2503.09556/) (2025)
- [From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion](https://mlsanity.com/paper/2204.03842/) (2022)
- [3D Face Reconstruction Using A Spectral-Based Graph Convolution Encoder](https://mlsanity.com/paper/2403.05218/) (2024)
- [An Augmentation-based Model Re-adaptation Framework for Robust Image Segmentation](https://mlsanity.com/paper/2409.09530/) (2024)
- [OFER: Occluded Face Expression Reconstruction](https://mlsanity.com/paper/2410.21629/) (2024)
- [3D Face Reconstruction From Radar Images](https://mlsanity.com/paper/2412.02403/) (2024)
- [Hierarchical MLANet: Multi-level Attention for 3D Face Reconstruction From Single Images](https://mlsanity.com/paper/2509.10024/) (2025)
- [MSMA: Multi-Scale Feature Fusion For Multi-Attribute 3D Face Reconstruction From Unconstrained Images](https://mlsanity.com/paper/2509.11763/) (2025)

## Abstract (arXiv)

In this work, we aim to enhance model-based face reconstruction by avoiding fitting the model to outliers, i.e. regions that cannot be well-expressed by the model such as occluders or make-up. The core challenge for localizing outliers is that they are highly variable and difficult to annotate. To overcome this challenging problem, we introduce a joint Face-autoencoder and outlier segmentation approach (FOCUS).In particular, we exploit the fact that the outliers cannot be fitted well by the face model and hence can be localized well given a high-quality model fitting. The main challenge is that the model fitting and the outlier segmentation are mutually dependent on each other, and need to be inferred jointly. We resolve this chicken-and-egg problem with an EM-type training strategy, where a face autoencoder is trained jointly with an outlier segmentation network. This leads to a synergistic effect, in which the segmentation network prevents the face encoder from fitting to the outliers, enhancing the reconstruction quality. The improved 3D face reconstruction, in turn, enables the segmentation network to better predict the outliers. To resolve the ambiguity between outliers and regions that are difficult to fit, such as eyebrows, we build a statistical prior from synthetic data that measures the systematic bias in model fitting. Experiments on the NoW testset demonstrate that FOCUS achieves SOTA 3D face reconstruction performance among all baselines that are trained without 3D annotation. Moreover, our results on CelebA-HQ and the AR database show that the segmentation network can localize occluders accurately despite being trained without any segmentation annotation.