# Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance

arXiv: https://arxiv.org/abs/1806.06503 · HTML: https://mlsanity.com/paper/1806.06503/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2018.

## Built on by (most cited first, 30 of 58)

- [RigNeRF: Fully Controllable Neural 3D Portraits](https://mlsanity.com/paper/2206.06481/) (2022)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [3D GAN Inversion for Controllable Portrait Image Animation](https://mlsanity.com/paper/2203.13441/) (2022)
- [OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wild](https://mlsanity.com/paper/2107.14480/) (2021)
- [High-Fidelity and Freely Controllable Talking Head Video Generation](https://mlsanity.com/paper/2304.10168/) (2023)
- [Unsupervised Learning of Group Invariant and Equivariant Representations](https://mlsanity.com/paper/2202.07559/) (2022)
- [FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation](https://mlsanity.com/paper/2108.04913/) (2021)
- [Unsupervised Human Pose Estimation through Transforming Shape Templates](https://mlsanity.com/paper/2105.04154/) (2021)
- [ICON: Learning Regular Maps Through Inverse Consistency](https://mlsanity.com/paper/2105.04459/) (2021)
- [Learning Facial Representations from the Cycle-consistency of Face](https://mlsanity.com/paper/2108.03427/) (2021)
- [FedDis: Disentangled Federated Learning for Unsupervised Brain Pathology Segmentation](https://mlsanity.com/paper/2103.03705/) (2021)
- [Unsupervised Image Representation Learning with Deep Latent Particles](https://mlsanity.com/paper/2205.15821/) (2022)
- [Learning monocular 3D reconstruction of articulated categories from motion](https://mlsanity.com/paper/2103.16352/) (2021)
- [Part-based Face Recognition with Vision Transformers](https://mlsanity.com/paper/2212.00057/) (2022)
- [Pose-disentangled Contrastive Learning for Self-supervised Facial Representation](https://mlsanity.com/paper/2211.13490/) (2022)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos](https://mlsanity.com/paper/2402.03723/) (2024)
- [Image-to-Video Generation via 3D Facial Dynamics](https://mlsanity.com/paper/2105.14678/) (2021)
- [Relightable Neural Actor with Intrinsic Decomposition and Pose Control](https://mlsanity.com/paper/2312.11587/) (2023)
- [A Generative Model of Symmetry Transformations](https://mlsanity.com/paper/2403.01946/) (2024)
- [Learning Canonical 3D Object Representation for Fine-Grained Recognition](https://mlsanity.com/paper/2108.04628/) (2021)
- [DisUnknown: Distilling Unknown Factors for Disentanglement Learning](https://mlsanity.com/paper/2109.08090/) (2021)
- [What do we learn? Debunking the Myth of Unsupervised Outlier Detection](https://mlsanity.com/paper/2206.03698/) (2022)
- [SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimation](https://mlsanity.com/paper/2405.18322/) (2024)
- [NODER: Image Sequence Regression Based on Neural Ordinary Differential Equations](https://mlsanity.com/paper/2407.13241/) (2024)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints](https://mlsanity.com/paper/2103.15812/) (2021)
- [Contrastive Learning of Person-independent Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2403.03400/) (2024)
- [MICS : Multi-steps, Inverse Consistency and Symmetric deep learning registration network](https://mlsanity.com/paper/2111.12123/) (2021)

## Abstract (arXiv)

In this work we introduce Deforming Autoencoders, a generative model for images that disentangles shape from appearance in an unsupervised manner. As in the deformable template paradigm, shape is represented as a deformation between a canonical coordinate system (`template') and an observed image, while appearance is modeled in `canonical', template, coordinates, thus discarding variability due to deformations. We introduce novel techniques that allow this approach to be deployed in the setting of autoencoders and show that this method can be used for unsupervised group-wise image alignment. We show experiments with expression morphing in humans, hands, and digits, face manipulation, such as shape and appearance interpolation, as well as unsupervised landmark localization. A more powerful form of unsupervised disentangling becomes possible in template coordinates, allowing us to successfully decompose face images into shading and albedo, and further manipulate face images.