# Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz

arXiv: https://arxiv.org/abs/1712.02859 · HTML: https://mlsanity.com/paper/1712.02859/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2017.

## Built on by (most cited first, 30 of 91)

- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset](https://mlsanity.com/paper/2203.14057/) (2022)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [MoFaNeRF: Morphable Facial Neural Radiance Field](https://mlsanity.com/paper/2112.02308/) (2021)
- [PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360circ](https://mlsanity.com/paper/2303.13071/) (2023)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [PRIOR: Prototype Representation Joint Learning from Medical Images and Reports](https://mlsanity.com/paper/2307.12577/) (2023)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery](https://mlsanity.com/paper/2304.05690/) (2023)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [Learning Disentangled Avatars with Hybrid 3D Representations](https://mlsanity.com/paper/2309.06441/) (2023)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field](https://mlsanity.com/paper/2309.17128/) (2023)
- [FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction](https://mlsanity.com/paper/2111.01082/) (2021)
- [Text and Image Guided 3D Avatar Generation and Manipulation](https://mlsanity.com/paper/2202.06079/) (2022)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/2105.07474/) (2021)
- [Deep Learning-based Face Super-Resolution: A Survey](https://mlsanity.com/paper/2101.03749/) (2021)
- [Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation](https://mlsanity.com/paper/2205.03962/) (2022)
- [Riggable 3D Face Reconstruction via In-Network Optimization](https://mlsanity.com/paper/2104.03493/) (2021)
- [GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations](https://mlsanity.com/paper/2409.11951/) (2024)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)

## Abstract (arXiv)

The reconstruction of dense 3D models of face geometry and appearance from a single image is highly challenging and ill-posed. To constrain the problem, many approaches rely on strong priors, such as parametric face models learned from limited 3D scan data. However, prior models restrict generalization of the true diversity in facial geometry, skin reflectance and illumination. To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model. Our multi-level face model combines the advantage of 3D Morphable Models for regularization with the out-of-space generalization of a learned corrective space. We train end-to-end on in-the-wild images without dense annotations by fusing a convolutional encoder with a differentiable expert-designed renderer and a self-supervised training loss, both defined at multiple detail levels. Our approach compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz.