# Learning Free-Form Deformation for 3D Face Reconstruction from In-The-Wild Images

arXiv: https://arxiv.org/abs/2105.14857 · HTML: https://mlsanity.com/paper/2105.14857/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression](https://mlsanity.com/paper/1703.07834/) (2017)
- [DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image](https://mlsanity.com/paper/1708.04672/) (2017)
- [Dense Face Alignment](https://mlsanity.com/paper/1709.01442/) (2017)
- [Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders](https://mlsanity.com/paper/1904.03525/) (2019)
- [Learning Free-Form Deformations for 3D Object Reconstruction](https://mlsanity.com/paper/1803.10932/) (2018)
- [Faster Than Real-time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Poses](https://mlsanity.com/paper/1707.05653/) (2017)
- [Learning Dense Facial Correspondences in Unconstrained Images](https://mlsanity.com/paper/1709.00536/) (2017)

## Built on by (most cited first, 7 of 7)

- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [SPARK: Self-supervised Personalized Real-time Monocular Face Capture](https://mlsanity.com/paper/2409.07984/) (2024)
- [Representing 3D Faces with Learnable B-Spline Volumes](https://mlsanity.com/paper/2604.12894/) (2026)
- [Registration-Free Learnable Multi-View Capture of Faces in Dense Semantic Correspondence](https://mlsanity.com/paper/2605.01450/) (2026)
- [OFER: Occluded Face Expression Reconstruction](https://mlsanity.com/paper/2410.21629/) (2024)
- [FaceSnap: Real-Time Personalized Lightstage Facial Performance Capture](https://mlsanity.com/paper/2608.31033/) (2026)

## Abstract (arXiv)

The 3D Morphable Model (3DMM), which is a Principal Component Analysis (PCA) based statistical model that represents a 3D face using linear basis functions, has shown promising results for reconstructing 3D faces from single-view in-the-wild images. However, 3DMM has restricted representation power due to the limited number of 3D scans and the global linear basis. To address the limitations of 3DMM, we propose a straightforward learning-based method that reconstructs a 3D face mesh through Free-Form Deformation (FFD) for the first time. FFD is a geometric modeling method that embeds a reference mesh within a parallelepiped grid and deforms the mesh by moving the sparse control points of the grid. As FFD is based on mathematically defined basis functions, it has no limitation in representation power. Thus, we can recover accurate 3D face meshes by estimating appropriate deviation of control points as deformation parameters. Although both 3DMM and FFD are parametric models, it is difficult to predict the effect of the 3DMM parameters on the face shape, while the deformation parameters of FFD are interpretable in terms of their effect on the final shape of the mesh. This practical advantage of FFD allows the resulting mesh and control points to serve as a good starting point for 3D face modeling, in that ordinary users can fine-tune the mesh by using widely available 3D software tools. Experiments on multiple datasets demonstrate how our method successfully estimates the 3D face geometry and facial expressions from 2D face images, achieving comparable performance to the state-of-the-art methods.