# 3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow

arXiv: https://arxiv.org/abs/2404.09819 · HTML: https://mlsanity.com/paper/2404.09819/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 30 of 38)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [PointAvatar: Deformable Point-based Head Avatars from Videos](https://mlsanity.com/paper/2212.08377/) (2022)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](https://mlsanity.com/paper/1910.12240/) (2019)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [CelebV-Text: A Large-Scale Facial Text-Video Dataset](https://mlsanity.com/paper/2303.14717/) (2023)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency](https://mlsanity.com/paper/2007.12494/) (2020)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images](https://mlsanity.com/paper/2302.14434/) (2023)
- [STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection](https://mlsanity.com/paper/2306.02763/) (2023)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://mlsanity.com/paper/2203.09729/) (2022)

## Built on by (most cited first, 20 of 20)

- [CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models](https://mlsanity.com/paper/2412.12093/) (2024)
- [Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction](https://mlsanity.com/paper/2505.00615/) (2025)
- [MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatars](https://mlsanity.com/paper/2510.12785/) (2025)
- [SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians](https://mlsanity.com/paper/2504.12292/) (2025)
- [Generating the Past, Present and Future from a Motion-Blurred Image](https://mlsanity.com/paper/2512.19817/) (2025)
- [ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation](https://mlsanity.com/paper/2607.06555/) (2026)
- [PrismAvatar: Real-time animated 3D neural head avatars on edge devices](https://mlsanity.com/paper/2502.07030/) (2025)
- [SemanticFace: Semantic Facial Action Estimation via Semantic Distillation in Interpretable Space](https://mlsanity.com/paper/2603.14827/) (2026)
- [PhysHead: Simulation-Ready Gaussian Head Avatars](https://mlsanity.com/paper/2604.06467/) (2026)
- [Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image](https://mlsanity.com/paper/2512.17773/) (2025)
- [Feed-forward Gaussian Registration for Head Avatar Creation and Editing](https://mlsanity.com/paper/2603.15811/) (2026)
- [Representing 3D Faces with Learnable B-Spline Volumes](https://mlsanity.com/paper/2604.12894/) (2026)
- [Registration-Free Learnable Multi-View Capture of Faces in Dense Semantic Correspondence](https://mlsanity.com/paper/2605.01450/) (2026)
- [High-Fidelity Single-Image Head Modeling with Industry-Grade Topology](https://mlsanity.com/paper/2605.04524/) (2026)
- [Improving Facial Rig Semantics for Tracking and Retargeting](https://mlsanity.com/paper/2508.08429/) (2025)
- [Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks](https://mlsanity.com/paper/2511.02830/) (2025)
- [GeoFace: Consistent Multi-View Face Generation with Geometry-Constrained Diffusion](https://mlsanity.com/paper/2606.27659/) (2026)
- [Enhancing Facial Expression Recognition in Head-Mounted Displays with Synthetic Data](https://mlsanity.com/paper/2607.04490/) (2026)
- [ViDS: Video Diffusion Shader using 3D Face Tracking](https://mlsanity.com/paper/2607.24124/) (2026)
- [RealDenseFace: Real-time Monocular 3D Face Reconstruction from Dense UV-space Priors](https://mlsanity.com/paper/2608.09238/) (2026)

## Abstract (arXiv)

When working with 3D facial data, improving fidelity and avoiding the uncanny valley effect is critically dependent on accurate 3D facial performance capture. Because such methods are expensive and due to the widespread availability of 2D videos, recent methods have focused on how to perform monocular 3D face tracking. However, these methods often fall short in capturing precise facial movements due to limitations in their network architecture, training, and evaluation processes. Addressing these challenges, we propose a novel face tracker, FlowFace, that introduces an innovative 2D alignment network for dense per-vertex alignment. Unlike prior work, FlowFace is trained on high-quality 3D scan annotations rather than weak supervision or synthetic data. Our 3D model fitting module jointly fits a 3D face model from one or many observations, integrating existing neutral shape priors for enhanced identity and expression disentanglement and per-vertex deformations for detailed facial feature reconstruction. Additionally, we propose a novel metric and benchmark for assessing tracking accuracy. Our method exhibits superior performance on both custom and publicly available benchmarks. We further validate the effectiveness of our tracker by generating high-quality 3D data from 2D videos, which leads to performance gains on downstream tasks.