# FML: Face Model Learning from Videos

arXiv: https://arxiv.org/abs/1812.07603 · HTML: https://mlsanity.com/paper/1812.07603/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2018.

## Built on by (most cited first, 30 of 78)

- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [NOFA: NeRF-based One-shot Facial Avatar Reconstruction](https://mlsanity.com/paper/2307.03441/) (2023)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2304.01436/) (2023)
- [Learning Disentangled Avatars with Hybrid 3D Representations](https://mlsanity.com/paper/2309.06441/) (2023)
- [FDNeRF: Few-shot Dynamic Neural Radiance Fields for Face Reconstruction and Expression Editing](https://mlsanity.com/paper/2208.05751/) (2022)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video](https://mlsanity.com/paper/2211.12782/) (2022)
- [Practical Face Reconstruction via Differentiable Ray Tracing](https://mlsanity.com/paper/2101.05356/) (2021)
- [Text and Image Guided 3D Avatar Generation and Manipulation](https://mlsanity.com/paper/2202.06079/) (2022)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/2105.07474/) (2021)
- [Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation](https://mlsanity.com/paper/2205.03962/) (2022)
- [Riggable 3D Face Reconstruction via In-Network Optimization](https://mlsanity.com/paper/2104.03493/) (2021)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction](https://mlsanity.com/paper/2106.03021/) (2021)
- [Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry](https://mlsanity.com/paper/2110.09772/) (2021)
- [Facial Geometric Detail Recovery via Implicit Representation](https://mlsanity.com/paper/2203.09692/) (2022)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [PV3D: A 3D Generative Model for Portrait Video Generation](https://mlsanity.com/paper/2212.06384/) (2022)
- [Learning Compositional Representation for 4D Captures with Neural ODE](https://mlsanity.com/paper/2103.08271/) (2021)
- [3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation](https://mlsanity.com/paper/2209.05434/) (2022)
- [SUPR: A Sparse Unified Part-Based Human Representation](https://mlsanity.com/paper/2210.13861/) (2022)
- [Learning a 3D Morphable Face Reflectance Model from Low-cost Data](https://mlsanity.com/paper/2303.11686/) (2023)

## Abstract (arXiv)

Monocular image-based 3D reconstruction of faces is a long-standing problem in computer vision. Since image data is a 2D projection of a 3D face, the resulting depth ambiguity makes the problem ill-posed. Most existing methods rely on data-driven priors that are built from limited 3D face scans. In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces. Our face model is learned using only corpora of in-the-wild video clips collected from the Internet. This virtually endless source of training data enables learning of a highly general 3D face model. In order to achieve this, we propose a novel multi-frame consistency loss that ensures consistent shape and appearance across multiple frames of a subject's face, thus minimizing depth ambiguity. At test time we can use an arbitrary number of frames, so that we can perform both monocular as well as multi-frame reconstruction.