# Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1609.06536 · HTML: https://mlsanity.com/paper/1609.06536/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2016.

## Built on by (most cited first, 14 of 14)

- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [Practical Face Reconstruction via Differentiable Ray Tracing](https://mlsanity.com/paper/2101.05356/) (2021)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction](https://mlsanity.com/paper/2106.03021/) (2021)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar](https://mlsanity.com/paper/2311.13655/) (2023)
- [Learning Audio-Driven Viseme Dynamics for 3D Face Animation](https://mlsanity.com/paper/2301.06059/) (2023)
- [SPARK: Self-supervised Personalized Real-time Monocular Face Capture](https://mlsanity.com/paper/2409.07984/) (2024)
- [AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction](https://mlsanity.com/paper/2304.13115/) (2023)
- [High-Quality Real Time Facial Capture Based on Single Camera](https://mlsanity.com/paper/2111.07556/) (2021)
- [Improved Detection of Face Presentation Attacks Using Image Decomposition](https://mlsanity.com/paper/2103.12201/) (2021)
- [SpecTracle: Wearable Facial Motion Tracking from Unobtrusive Peripheral Cameras](https://mlsanity.com/paper/2308.07502/) (2023)
- [FDLS: A Deep Learning Approach to Production Quality, Controllable, and Retargetable Facial Performances](https://mlsanity.com/paper/2309.14897/) (2023)
- [Improving Facial Rig Semantics for Tracking and Retargeting](https://mlsanity.com/paper/2508.08429/) (2025)

## Abstract (arXiv)

We present a real-time deep learning framework for video-based facial performance capture -- the dense 3D tracking of an actor's face given a monocular video. Our pipeline begins with accurately capturing a subject using a high-end production facial capture pipeline based on multi-view stereo tracking and artist-enhanced animations. With 5-10 minutes of captured footage, we train a convolutional neural network to produce high-quality output, including self-occluded regions, from a monocular video sequence of that subject. Since this 3D facial performance capture is fully automated, our system can drastically reduce the amount of labor involved in the development of modern narrative-driven video games or films involving realistic digital doubles of actors and potentially hours of animated dialogue per character. We compare our results with several state-of-the-art monocular real-time facial capture techniques and demonstrate compelling animation inference in challenging areas such as eyes and lips.