# Demystifying Inductive Biases for β-VAE Based Architectures

arXiv: https://arxiv.org/abs/2102.06822 · HTML: https://mlsanity.com/paper/2102.06822/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness](https://mlsanity.com/paper/1811.00007/) (2018)
- [Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis](https://mlsanity.com/paper/1912.05237/) (2019)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation](https://mlsanity.com/paper/2010.14766/) (2020)
- [Unsupervised Model Selection for Variational Disentangled Representation Learning](https://mlsanity.com/paper/1905.12614/) (2019)
- [Group invariance principles for causal generative models](https://mlsanity.com/paper/1705.02212/) (2017)
- [A theory of independent mechanisms for extrapolation in generative models](https://mlsanity.com/paper/2004.00184/) (2020)
- [Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1909.05063/) (2019)

## Built on by (most cited first, 5 of 5)

- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [Overlooked Implications of the Reconstruction Loss for VAE Disentanglement](https://mlsanity.com/paper/2202.13341/) (2022)
- [Latent Noise Segmentation: How Neural Noise Leads to the Emergence of Segmentation and Grouping](https://mlsanity.com/paper/2309.16515/) (2023)
- [Unscented Autoencoder](https://mlsanity.com/paper/2306.05256/) (2023)
- [Disentangled Representation Learning Using (β-)VAE and GAN](https://mlsanity.com/paper/2208.04549/) (2022)

## Abstract (arXiv)

The performance of β-Variational-Autoencoders (β-VAEs) and their variants on learning semantically meaningful, disentangled representations is unparalleled. On the other hand, there are theoretical arguments suggesting the impossibility of unsupervised disentanglement. In this work, we shed light on the inductive bias responsible for the success of VAE-based architectures. We show that in classical datasets the structure of variance, induced by the generating factors, is conveniently aligned with the latent directions fostered by the VAE objective. This builds the pivotal bias on which the disentangling abilities of VAEs rely. By small, elaborate perturbations of existing datasets, we hide the convenient correlation structure that is easily exploited by a variety of architectures. To demonstrate this, we construct modified versions of standard datasets in which (i) the generative factors are perfectly preserved; (ii) each image undergoes a mild transformation causing a small change of variance; (iii) the leading VAE-based disentanglement architectures fail to produce disentangled representations whilst the performance of a non-variational method remains unchanged. The construction of our modifications is nontrivial and relies on recent progress on mechanistic understanding of β-VAEs and their connection to PCA. We strengthen that connection by providing additional insights that are of stand-alone interest.