# Real-Time Facial Segmentation and Performance Capture from RGB Input

arXiv: https://arxiv.org/abs/1604.02647 · HTML: https://mlsanity.com/paper/1604.02647/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2016.

## Built on by (most cited first, 13 of 13)

- [Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videos](https://mlsanity.com/paper/2207.11094/) (2022)
- [Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation](https://mlsanity.com/paper/2106.09614/) (2021)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Mask-FPAN: Semi-Supervised Face Parsing in the Wild With De-Occlusion and UV GAN](https://mlsanity.com/paper/2212.09098/) (2022)
- [Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets](https://mlsanity.com/paper/2205.06218/) (2022)
- [Segmentation-Reconstruction-Guided Facial Image De-occlusion](https://mlsanity.com/paper/2112.08022/) (2021)
- [Generating Diverse 3D Reconstructions from a Single Occluded Face Image](https://mlsanity.com/paper/2112.00879/) (2021)
- [FaceOcc: A Diverse, High-quality Face Occlusion Dataset for Human Face Extraction](https://mlsanity.com/paper/2201.08425/) (2022)
- [Reconstructing Recognizable 3D Face Shapes based on 3D Morphable Models](https://mlsanity.com/paper/2104.03515/) (2021)
- [REFA: Real-time Egocentric Facial Animations for Virtual Reality](https://mlsanity.com/paper/2601.03507/) (2026)
- [A Facial Feature Discovery Framework for Race Classification Using Deep Learning](https://mlsanity.com/paper/2104.02471/) (2021)
- [Hierarchical MLANet: Multi-level Attention for 3D Face Reconstruction From Single Images](https://mlsanity.com/paper/2509.10024/) (2025)
- [MSMA: Multi-Scale Feature Fusion For Multi-Attribute 3D Face Reconstruction From Unconstrained Images](https://mlsanity.com/paper/2509.11763/) (2025)

## Abstract (arXiv)

We introduce the concept of unconstrained real-time 3D facial performance capture through explicit semantic segmentation in the RGB input. To ensure robustness, cutting edge supervised learning approaches rely on large training datasets of face images captured in the wild. While impressive tracking quality has been demonstrated for faces that are largely visible, any occlusion due to hair, accessories, or hand-to-face gestures would result in significant visual artifacts and loss of tracking accuracy. The modeling of occlusions has been mostly avoided due to its immense space of appearance variability. To address this curse of high dimensionality, we perform tracking in unconstrained images assuming non-face regions can be fully masked out. Along with recent breakthroughs in deep learning, we demonstrate that pixel-level facial segmentation is possible in real-time by repurposing convolutional neural networks designed originally for general semantic segmentation. We develop an efficient architecture based on a two-stream deconvolution network with complementary characteristics, and introduce carefully designed training samples and data augmentation strategies for improved segmentation accuracy and robustness. We adopt a state-of-the-art regression-based facial tracking framework with segmented face images as training, and demonstrate accurate and uninterrupted facial performance capture in the presence of extreme occlusion and even side views. Furthermore, the resulting segmentation can be directly used to composite partial 3D face models on the input images and enable seamless facial manipulation tasks, such as virtual make-up or face replacement.