# Disentangling factors of variation in deep representations using adversarial training

arXiv: https://arxiv.org/abs/1611.03383 · HTML: https://mlsanity.com/paper/1611.03383/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2016.

## Built on by (most cited first, 30 of 87)

- [ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis](https://mlsanity.com/paper/2108.08827/) (2021)
- [Factorizing Knowledge in Neural Networks](https://mlsanity.com/paper/2207.03337/) (2022)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [Style Normalization and Restitution for Domain Generalization and Adaptation](https://mlsanity.com/paper/2101.00588/) (2021)
- [Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models](https://mlsanity.com/paper/2105.08127/) (2021)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [Learning Fair Representation via Distributional Contrastive Disentanglement](https://mlsanity.com/paper/2206.08743/) (2022)
- [StyleFusion: A Generative Model for Disentangling Spatial Segments](https://mlsanity.com/paper/2107.07437/) (2021)
- [Source-free unsupervised domain adaptation for cross-modality abdominal multi-organ segmentation](https://mlsanity.com/paper/2111.12221/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [A Unified Conditional Disentanglement Framework for Multimodal Brain MR Image Translation](https://mlsanity.com/paper/2101.05434/) (2021)
- [Transformers with Competitive Ensembles of Independent Mechanisms](https://mlsanity.com/paper/2103.00336/) (2021)
- [Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity](https://mlsanity.com/paper/2111.02338/) (2021)
- [Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction](https://mlsanity.com/paper/2111.06636/) (2021)
- [Decoupling Local and Global Representations of Time Series](https://mlsanity.com/paper/2202.02262/) (2022)
- [How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processing](https://mlsanity.com/paper/2205.10095/) (2022)
- [Scaling-up Disentanglement for Image Translation](https://mlsanity.com/paper/2103.14017/) (2021)
- [Decompose to Adapt: Cross-domain Object Detection via Feature Disentanglement](https://mlsanity.com/paper/2201.01929/) (2022)
- [UDoc-GAN: Unpaired Document Illumination Correction with Background Light Prior](https://mlsanity.com/paper/2210.08216/) (2022)
- [Behavior-Driven Synthesis of Human Dynamics](https://mlsanity.com/paper/2103.04677/) (2021)
- [A Contrastive Objective for Learning Disentangled Representations](https://mlsanity.com/paper/2203.11284/) (2022)
- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks](https://mlsanity.com/paper/2112.10082/) (2021)
- [Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement](https://mlsanity.com/paper/2102.05185/) (2021)
- [Disentangle then Parse:Night-time Semantic Segmentation with Illumination Disentanglement](https://mlsanity.com/paper/2307.09362/) (2023)
- [Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion](https://mlsanity.com/paper/2312.03475/) (2023)
- [Identity-aware Facial Expression Recognition in Compressed Video](https://mlsanity.com/paper/2101.00317/) (2021)
- [Learning Disentangled Behaviour Patterns for Wearable-based Human Activity Recognition](https://mlsanity.com/paper/2202.07260/) (2022)
- [Weak Adaptation Learning -- Addressing Cross-domain Data Insufficiency with Weak Annotator](https://mlsanity.com/paper/2102.07358/) (2021)
- [Disentangled Generative Models for Robust Prediction of System Dynamics](https://mlsanity.com/paper/2108.11684/) (2021)

## Abstract (arXiv)

We introduce a conditional generative model for learning to disentangle the hidden factors of variation within a set of labeled observations, and separate them into complementary codes. One code summarizes the specified factors of variation associated with the labels. The other summarizes the remaining unspecified variability. During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class. Examples of such observations include images of a set of labeled objects captured at different viewpoints, or recordings of set of speakers dictating multiple phrases. In both instances, the intra-class diversity is the source of the unspecified factors of variation: each object is observed at multiple viewpoints, and each speaker dictates multiple phrases. Learning to disentangle the specified factors from the unspecified ones becomes easier when strong supervision is possible. Suppose that during training, we have access to pairs of images, where each pair shows two different objects captured from the same viewpoint. This source of alignment allows us to solve our task using existing methods. However, labels for the unspecified factors are usually unavailable in realistic scenarios where data acquisition is not strictly controlled. We address the problem of disentanglement in this more general setting by combining deep convolutional autoencoders with a form of adversarial training. Both factors of variation are implicitly captured in the organization of the learned embedding space, and can be used for solving single-image analogies. Experimental results on synthetic and real datasets show that the proposed method is capable of generalizing to unseen classes and intra-class variabilities.