# Instant Multi-View Head Capture through Learnable Registration

arXiv: https://arxiv.org/abs/2306.07437 · HTML: https://mlsanity.com/paper/2306.07437/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 92 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research](https://mlsanity.com/paper/1911.05063/) (2019)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [Deep Implicit Templates for 3D Shape Representation](https://mlsanity.com/paper/2011.14565/) (2020)
- [FML: Face Model Learning from Videos](https://mlsanity.com/paper/1812.07603/) (2018)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)

## Built on by (most cited first, 30 of 34)

- [Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2312.03704/) (2023)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [HHAvatar: Gaussian Head Avatar with Dynamic Hairs](https://mlsanity.com/paper/2312.03029/) (2023)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow](https://mlsanity.com/paper/2404.09819/) (2024)
- [ILSH: The Imperial Light-Stage Head Dataset for Human Head View Synthesis](https://mlsanity.com/paper/2310.03952/) (2023)
- [Category-Agnostic Neural Object Rigging](https://mlsanity.com/paper/2505.20283/) (2025)
- [Topo4D: Topology-Preserving Gaussian Splatting for High-Fidelity 4D Head Capture](https://mlsanity.com/paper/2406.00440/) (2024)
- [GRAPE: Generalizable and Robust Multi-view Facial Capture](https://mlsanity.com/paper/2407.10193/) (2024)
- [SEREP: Semantic Facial Expression Representation for Robust In-the-Wild Capture and Retargeting](https://mlsanity.com/paper/2412.14371/) (2024)
- [LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs](https://mlsanity.com/paper/2506.11148/) (2025)
- [Express4D: Expressive, Friendly, and Extensible 4D Facial Motion Generation Benchmark](https://mlsanity.com/paper/2508.12438/) (2025)
- [VGGTFace: Topologically Consistent Facial Geometry Reconstruction in the Wild](https://mlsanity.com/paper/2511.20366/) (2025)
- [Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection](https://mlsanity.com/paper/2405.20117/) (2024)
- [Pixels2Points: Fusing 2D and 3D Features for Facial Skin Segmentation](https://mlsanity.com/paper/2504.19718/) (2025)
- [Democratizing the Creation of Animatable Facial Avatars](https://mlsanity.com/paper/2401.16534/) (2024)
- [Reconstructing Topology-Consistent Face Mesh by Volume Rendering from Multi-View Images](https://mlsanity.com/paper/2404.05606/) (2024)
- [PaCo-FR: Patch-Pixel Aligned End-to-End Codebook Learning for Facial Representation Pre-training](https://mlsanity.com/paper/2508.09691/) (2025)
- [Free-Boundary Quasiconformal Maps via a Least-squares Operator in Diffeomorphism Optimization](https://mlsanity.com/paper/2511.11679/) (2025)
- [Feed-forward Gaussian Registration for Head Avatar Creation and Editing](https://mlsanity.com/paper/2603.15811/) (2026)
- [Representing 3D Faces with Learnable B-Spline Volumes](https://mlsanity.com/paper/2604.12894/) (2026)
- [Registration-Free Learnable Multi-View Capture of Faces in Dense Semantic Correspondence](https://mlsanity.com/paper/2605.01450/) (2026)
- [High-Fidelity Single-Image Head Modeling with Industry-Grade Topology](https://mlsanity.com/paper/2605.04524/) (2026)
- [Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration](https://mlsanity.com/paper/2403.02629/) (2024)
- [epsilon-Mesh Attack: A Surface-based Adversarial Point Cloud Attack for Facial Expression Recognition](https://mlsanity.com/paper/2403.06661/) (2024)
- [3D2M Dataset: A 3-Dimension diverse Mesh Dataset](https://mlsanity.com/paper/2410.07415/) (2024)
- [OFER: Occluded Face Expression Reconstruction](https://mlsanity.com/paper/2410.21629/) (2024)
- [Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture](https://mlsanity.com/paper/2512.16397/) (2025)
- [TOPOS: High-Fidelity and Efficient Industry-Grade 3D Head Generation](https://mlsanity.com/paper/2605.14594/) (2026)
- [CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation](https://mlsanity.com/paper/2605.29316/) (2026)

## Abstract (arXiv)

Existing methods for capturing datasets of 3D heads in dense semantic correspondence are slow, and commonly address the problem in two separate steps; multi-view stereo (MVS) reconstruction followed by non-rigid registration. To simplify this process, we introduce TEMPEH (Towards Estimation of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads in dense correspondence from calibrated multi-view images. Registering datasets of 3D scans typically requires manual parameter tuning to find the right balance between accurately fitting the scans surfaces and being robust to scanning noise and outliers. Instead, we propose to jointly register a 3D head dataset while training TEMPEH. Specifically, during training we minimize a geometric loss commonly used for surface registration, effectively leveraging TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric feature representation that samples and fuses features from each view using camera calibration information. To account for partial occlusions and a large capture volume that enables head movements, we use view- and surface-aware feature fusion, and a spatial transformer-based head localization module, respectively. We use raw MVS scans as supervision during training, but, once trained, TEMPEH directly predicts 3D heads in dense correspondence without requiring scans. Predicting one head takes about 0.3 seconds with a median reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art. This enables the efficient capture of large datasets containing multiple people and diverse facial motions. Code, model, and data are publicly available at https://tempeh.is.tue.mpg.de.