# Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model using Deep Non-Rigid Structure from Motion

arXiv: https://arxiv.org/abs/1904.11960 · HTML: https://mlsanity.com/paper/1904.11960/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2019.

## Built on by (most cited first, 23 of 23)

- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis](https://mlsanity.com/paper/2110.15678/) (2021)
- [Generative Occupancy Fields for 3D Surface-Aware Image Synthesis](https://mlsanity.com/paper/2111.00969/) (2021)
- [Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation](https://mlsanity.com/paper/2205.03962/) (2022)
- [De-rendering the World's Revolutionary Artefacts](https://mlsanity.com/paper/2104.03954/) (2021)
- [φ-SfT: Shape-from-Template with a Physics-Based Deformation Model](https://mlsanity.com/paper/2203.11938/) (2022)
- [To The Point: Correspondence-driven monocular 3D category reconstruction](https://mlsanity.com/paper/2106.05662/) (2021)
- [Learning monocular 3D reconstruction of articulated categories from motion](https://mlsanity.com/paper/2103.16352/) (2021)
- [Part-based Face Recognition with Vision Transformers](https://mlsanity.com/paper/2212.00057/) (2022)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [AR-NeRF: Unsupervised Learning of Depth and Defocus Effects from Natural Images with Aperture Rendering Neural Radiance Fields](https://mlsanity.com/paper/2206.06100/) (2022)
- [From Keypoints to Object Landmarks via Self-Training Correspondence: A novel approach to Unsupervised Landmark Discovery](https://mlsanity.com/paper/2205.15895/) (2022)
- [MI-NeRF: Learning a Single Face NeRF from Multiple Identities](https://mlsanity.com/paper/2403.19920/) (2024)
- [End-to-End Learning of Multi-category 3D Pose and Shape Estimation](https://mlsanity.com/paper/2112.10196/) (2021)
- [A Generative Framework for Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2309.08273/) (2023)
- [Unsupervised Landmark Discovery Using Consistency Guided Bottleneck](https://mlsanity.com/paper/2309.10518/) (2023)
- [Exploring Deep Registration Latent Spaces](https://mlsanity.com/paper/2107.11238/) (2021)
- [Piecewise Planar Hulls for Semi-Supervised Learning of 3D Shape and Pose from 2D Images](https://mlsanity.com/paper/2211.07491/) (2022)
- [Unsupervised Style-based Explicit 3D Face Reconstruction from Single Image](https://mlsanity.com/paper/2304.12455/) (2023)
- [OrcoDCS: An IoT-Edge Orchestrated Online Deep Compressed Sensing Framework](https://mlsanity.com/paper/2308.05757/) (2023)
- [Disentangled Human Body Representation Based on Unsupervised Semantic-Aware Learning](https://mlsanity.com/paper/2505.19049/) (2025)
- [KeyPointDiffuser: Unsupervised 3D Keypoint Learning via Latent Diffusion Models](https://mlsanity.com/paper/2512.03450/) (2025)

## Abstract (arXiv)

In this work we introduce Lifting Autoencoders, a generative 3D surface-based model of object categories. We bring together ideas from non-rigid structure from motion, image formation, and morphable models to learn a controllable, geometric model of 3D categories in an entirely unsupervised manner from an unstructured set of images. We exploit the 3D geometric nature of our model and use normal information to disentangle appearance into illumination, shading and albedo. We further use weak supervision to disentangle the non-rigid shape variability of human faces into identity and expression. We combine the 3D representation with a differentiable renderer to generate RGB images and append an adversarially trained refinement network to obtain sharp, photorealistic image reconstruction results. The learned generative model can be controlled in terms of interpretable geometry and appearance factors, allowing us to perform photorealistic image manipulation of identity, expression, 3D pose, and illumination properties.