# NestedVAE: Isolating Common Factors via Weak Supervision

arXiv: https://arxiv.org/abs/2002.11576 · HTML: https://mlsanity.com/paper/2002.11576/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 13 of 13)

- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Discover the Unknown Biased Attribute of an Image Classifier](https://mlsanity.com/paper/2104.14556/) (2021)
- [Gacs-Korner Common Information Variational Autoencoder](https://mlsanity.com/paper/2205.12239/) (2022)
- [Double InfoGAN for Contrastive Analysis](https://mlsanity.com/paper/2401.17776/) (2024)
- [VDSM: Unsupervised Video Disentanglement with State-Space Modeling and Deep Mixtures of Experts](https://mlsanity.com/paper/2103.07292/) (2021)
- [Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audio](https://mlsanity.com/paper/2205.05871/) (2022)
- [ARD-VAE: A Statistical Formulation to Find the Relevant Latent Dimensions of Variational Autoencoders](https://mlsanity.com/paper/2501.10901/) (2025)
- [SKILL-IL: Disentangling Skill and Knowledge in Multitask Imitation Learning](https://mlsanity.com/paper/2205.03130/) (2022)
- [Learning ABCs: Approximate Bijective Correspondence for isolating factors of variation with weak supervision](https://mlsanity.com/paper/2103.03240/) (2021)
- [Multimodal hierarchical Variational AutoEncoders with Factor Analysis latent space](https://mlsanity.com/paper/2207.09185/) (2022)
- [Comparing the information content of probabilistic representation spaces](https://mlsanity.com/paper/2405.21042/) (2024)
- [Symbolic Disentangled Representations for Images](https://mlsanity.com/paper/2412.19847/) (2024)
- [Differential Adjusted Parity for Learning Fair Representations](https://mlsanity.com/paper/2502.09765/) (2025)

## Abstract (arXiv)

Fair and unbiased machine learning is an important and active field of research, as decision processes are increasingly driven by models that learn from data. Unfortunately, any biases present in the data may be learned by the model, thereby inappropriately transferring that bias into the decision making process. We identify the connection between the task of bias reduction and that of isolating factors common between domains whilst encouraging domain specific invariance. To isolate the common factors we combine the theory of deep latent variable models with information bottleneck theory for scenarios whereby data may be naturally paired across domains and no additional supervision is required. The result is the Nested Variational AutoEncoder (NestedVAE). Two outer VAEs with shared weights attempt to reconstruct the input and infer a latent space, whilst a nested VAE attempts to reconstruct the latent representation of one image, from the latent representation of its paired image. In so doing, the nested VAE isolates the common latent factors/causes and becomes invariant to unwanted factors that are not shared between paired images. We also propose a new metric to provide a balanced method of evaluating consistency and classifier performance across domains which we refer to as the Adjusted Parity metric. An evaluation of NestedVAE on both domain and attribute invariance, change detection, and learning common factors for the prediction of biological sex demonstrates that NestedVAE significantly outperforms alternative methods.