# Single-Camera 3D Head Fitting for Mixed Reality Clinical Applications

arXiv: https://arxiv.org/abs/2109.02740 · HTML: https://mlsanity.com/paper/2109.02740/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Detailed Human Avatars from Monocular Video](https://mlsanity.com/paper/1808.01338/) (2018)
- [Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"](https://mlsanity.com/paper/1908.07201/) (2019)
- [Towards a complete 3D morphable model of the human head](https://mlsanity.com/paper/1911.08008/) (2019)
- [Video to Fully Automatic 3D Hair Model](https://mlsanity.com/paper/1809.04765/) (2018)
- [Sparse Photometric 3D Face Reconstruction Guided by Morphable Models](https://mlsanity.com/paper/1711.10870/) (2017)
- [HairNet: Single-View Hair Reconstruction using Convolutional Neural Networks](https://mlsanity.com/paper/1806.07467/) (2018)
- [High Accuracy Face Geometry Capture using a Smartphone Video](https://mlsanity.com/paper/2003.08583/) (2020)

## Built on by (most cited first, 2 of 2)

- [3 Dimensional Dense Reconstruction: A Review of Algorithms and Dataset](https://mlsanity.com/paper/2304.09371/) (2023)
- [Bridging the Ex-Vivo to In-Vivo Gap: Synthetic Priors for Monocular Depth Estimation in Specular Surgical Environments](https://mlsanity.com/paper/2512.23786/) (2025)

## Abstract (arXiv)

We address the problem of estimating the shape of a person's head, defined as the geometry of the complete head surface, from a video taken with a single moving camera, and determining the alignment of the fitted 3D head for all video frames, irrespective of the person's pose. 3D head reconstructions commonly tend to focus on perfecting the face reconstruction, leaving the scalp to a statistical approximation. Our goal is to reconstruct the head model of each person to enable future mixed reality applications. To do this, we recover a dense 3D reconstruction and camera information via structure-from-motion and multi-view stereo. These are then used in a new two-stage fitting process to recover the 3D head shape by iteratively fitting a 3D morphable model of the head with the dense reconstruction in canonical space and fitting it to each person's head, using both traditional facial landmarks and scalp features extracted from the head's segmentation mask. Our approach recovers consistent geometry for varying head shapes, from videos taken by different people, with different smartphones, and in a variety of environments from living rooms to outdoor spaces.