# Where and What? Examining Interpretable Disentangled Representations

arXiv: https://arxiv.org/abs/2104.05622 · HTML: https://mlsanity.com/paper/2104.05622/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 62 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [On the Fairness of Disentangled Representations](https://mlsanity.com/paper/1905.13662/) (2019)
- [Deep Convolutional Inverse Graphics Network](https://mlsanity.com/paper/1503.03167/) (2015)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [Domain Agnostic Learning with Disentangled Representations](https://mlsanity.com/paper/1904.12347/) (2019)
- [Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks](https://mlsanity.com/paper/1810.09536/) (2018)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](https://mlsanity.com/paper/1905.12506/) (2019)
- [Attribute2Image: Conditional Image Generation from Visual Attributes](https://mlsanity.com/paper/1512.00570/) (2015)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)

## Built on by (most cited first, 27 of 27)

- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2203.15799/) (2022)
- [Commutative Lie Group VAE for Disentanglement Learning](https://mlsanity.com/paper/2106.03375/) (2021)
- [Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement](https://mlsanity.com/paper/2402.09712/) (2024)
- [Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representations](https://mlsanity.com/paper/2112.02290/) (2021)
- [DyTed: Disentangled Representation Learning for Discrete-time Dynamic Graph](https://mlsanity.com/paper/2210.10592/) (2022)
- [Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedback](https://mlsanity.com/paper/2402.02346/) (2024)
- [DSVAE: Interpretable Disentangled Representation for Synthetic Speech Detection](https://mlsanity.com/paper/2304.03323/) (2023)
- [Seven challenges for harmonizing explainability requirements](https://mlsanity.com/paper/2108.05390/) (2021)
- [Modelling Direct Messaging Networks with Multiple Recipients for Cyber Deception](https://mlsanity.com/paper/2111.11932/) (2021)
- [Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors](https://mlsanity.com/paper/2210.01760/) (2022)
- [Predefined Prototypes for Intra-Class Separation and Disentanglement](https://mlsanity.com/paper/2406.16145/) (2024)
- [ChildPredictor: A Child Face Prediction Framework with Disentangled Learning](https://mlsanity.com/paper/2204.09962/) (2022)
- [A Survey on Leveraging Pre-trained Generative Adversarial Networks for Image Editing and Restoration](https://mlsanity.com/paper/2207.10309/) (2022)
- [Joint Quality Assessment and Example-Guided Image Processing by Disentangling Picture Appearance from Content](https://mlsanity.com/paper/2404.13484/) (2024)
- [Transferring disentangled representations: bridging the gap between synthetic and real images](https://mlsanity.com/paper/2409.18017/) (2024)
- [DiViD: Disentangled Video Diffusion for Static-Dynamic Factorization](https://mlsanity.com/paper/2507.13934/) (2025)
- [Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment](https://mlsanity.com/paper/2510.11112/) (2025)
- [Text-guided Feature Disentanglement for Cross-modal Gait Recognition](https://mlsanity.com/paper/2605.30784/) (2026)
- [MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning](https://mlsanity.com/paper/2607.16789/) (2026)
- [ContraFeat: Contrasting Deep Features for Semantic Discovery](https://mlsanity.com/paper/2212.07277/) (2022)
- [Explainability and Robustness of Deep Visual Classification Models](https://mlsanity.com/paper/2301.01343/) (2023)
- [Rethinking Disentanglement under Dependent Factors of Variation](https://mlsanity.com/paper/2408.07016/) (2024)
- [Independence Constrained Disentangled Representation Learning from Epistemological Perspective](https://mlsanity.com/paper/2409.02672/) (2024)
- [Disentanglement in Difference: Directly Learning Semantically Disentangled Representations by Maximizing Inter-Factor Differences](https://mlsanity.com/paper/2502.03123/) (2025)
- [Disentangled representations of microscopy images](https://mlsanity.com/paper/2506.20649/) (2025)

## Abstract (arXiv)

Capturing interpretable variations has long been one of the goals in disentanglement learning. However, unlike the independence assumption, interpretability has rarely been exploited to encourage disentanglement in the unsupervised setting. In this paper, we examine the interpretability of disentangled representations by investigating two questions: where to be interpreted and what to be interpreted? A latent code is easily to be interpreted if it would consistently impact a certain subarea of the resulting generated image. We thus propose to learn a spatial mask to localize the effect of each individual latent dimension. On the other hand, interpretability usually comes from latent dimensions that capture simple and basic variations in data. We thus impose a perturbation on a certain dimension of the latent code, and expect to identify the perturbation along this dimension from the generated images so that the encoding of simple variations can be enforced. Additionally, we develop an unsupervised model selection method, which accumulates perceptual distance scores along axes in the latent space. On various datasets, our models can learn high-quality disentangled representations without supervision, showing the proposed modeling of interpretability is an effective proxy for achieving unsupervised disentanglement.