# 3D Morphable Models as Spatial Transformer Networks

arXiv: https://arxiv.org/abs/1708.07199 · HTML: https://mlsanity.com/paper/1708.07199/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2017.

## Built on by (most cited first, 12 of 12)

- [Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement](https://mlsanity.com/paper/2106.11423/) (2021)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Weakly-Supervised Multi-Face 3D Reconstruction](https://mlsanity.com/paper/2101.02000/) (2021)
- [Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape](https://mlsanity.com/paper/2204.04379/) (2022)
- [Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes](https://mlsanity.com/paper/2102.01161/) (2021)
- [FaceGPT: Self-supervised Learning to Chat about 3D Human Faces](https://mlsanity.com/paper/2406.07163/) (2024)
- [Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection](https://mlsanity.com/paper/2405.20117/) (2024)
- [A Review of 3D Face Reconstruction From a Single Image](https://mlsanity.com/paper/2110.09299/) (2021)
- [Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks](https://mlsanity.com/paper/2305.19130/) (2023)
- [A lightweight 3D dense facial landmark estimation model from position map data](https://mlsanity.com/paper/2308.15170/) (2023)
- [Technical Report of 2023 ABO Fine-grained Semantic Segmentation Competition](https://mlsanity.com/paper/2310.00427/) (2023)
- [Vision-Language Modeling with Regularized Spatial Transformer Networks for All Weather Crosswind Landing of Aircraft](https://mlsanity.com/paper/2405.05574/) (2024)

## Abstract (arXiv)

In this paper, we show how a 3D Morphable Model (i.e. a statistical model of the 3D shape of a class of objects such as faces) can be used to spatially transform input data as a module (a 3DMM-STN) within a convolutional neural network. This is an extension of the original spatial transformer network in that we are able to interpret and normalise 3D pose changes and self-occlusions. The trained localisation part of the network is independently useful since it learns to fit a 3D morphable model to a single image. We show that the localiser can be trained using only simple geometric loss functions on a relatively small dataset yet is able to perform robust normalisation on highly uncontrolled images including occlusion, self-occlusion and large pose changes.